Importing archives into WordPress

I’m starting with the import process not because it is an exceptionally good place to start when preparing to move a site to WordPress but because it’s one of the few things I got right from the get-go when I transferred my first news site to WordPress.

The best way to import content into WordPress, in my experience, is using WordPress’s XML import.

WordPress’s XML files follow an easy-to-grasp but powerful structure that, in general, goes something like this:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:excerpt="http://wordpress.org/export/1.0/excerpt/"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:wp="http://wordpress.org/export/1.0/"
>

<channel>
	<title>My example site</title>
	<link>http://example.com/</link>
	<description></description>
	<pubDate>Thu, 28 May 2009 16:06:40 +0000</pubDate>
	<generator>http://wordpress.org/?v=2.7.1</generator>
	<language>en</language>
	<wp:wxr_version>1.0</wp:wxr_version>
	<wp:base_site_url>http://example.com/</wp:base_site_url>
	<wp:base_blog_url>http://example.com/</wp:base_blog_url>

	<item>
		<category domain="category" nicename="my-category"><![CDATA[My Category]]></category>
		<category domain="tag" nicename="my-tag"><![CDATA[My Tag]]></category>
		<title><![CDATA[My Post Title]]></title>
		<dc:creator><![CDATA[My Name]]></dc:creator>
		<link>http://example.com/2010/07/06/my-post-title/</link>
		<pubDate>Tue, 06 Jul 2010 10:51:32 +0000</pubDate><dc:creator><![CDATA[bdnoutdoors]]></dc:creator>
		<guid isPermaLink="false">http://example.com/?p=12345</guid>
		<description></description>
		<content:encoded><![CDATA[My post content.]]></content:encoded>
		<excerpt:encoded><![CDATA[My post excerpt.]]></excerpt:encoded>
		<wp:post_id>12345</wp:post_id>
		<wp:post_date>2010-07-06 10:51:32</wp:post_date>
		<wp:post_date_gmt>2010-07-06 10:51:32</wp:post_date_gmt>
		<wp:comment_status>open</wp:comment_status>
		<wp:ping_status>closed</wp:ping_status>
		<wp:post_name>my-post-title</wp:post_name>
		<wp:status>publish</wp:status>
		<wp:post_parent>0</wp:post_parent>
		<wp:menu_order>0</wp:menu_order>
		<wp:post_type>post</wp:post_type>
		<wp:post_password></wp:post_password>

		<wp:postmeta>
			<wp:meta_key>my_post_meta_key</wp:meta_key>
			<wp:meta_value>My Post Meta Value</wp:meta_value>
		</wp:postmeta>
	</item>

</channel>
</rss>

That’s a fairly simple usage of WordPress XML, and when we imported our content from the Bangor Daily News we did a lot more.

For example, we imported all our posts with a hidden post meta (_old_id) value of the article’s ID in our old CMS. Then, we used the CP Redirect plugin as a template for a new plugin to redirect people clicking on old links to the new URL.

We also found that using the <dc:creator> tag quickly overwhelmed us. As with any newspaper, there are thousands of people who have written just one or a few articles for us, and we didn’t want to create accounts for all of them. Instead, we created a whitelist of authors we wanted to come in as users — basically just BDN staff and frequent contributors and freelancers — and the rest of the posts came in with a default username and with the author’s name in a most meta field name _byline.

We don’t embed images in posts. Rather, we query for all images attached to the post and display them at the top of the post and in the sidebar (more about this in a later post). So we natively imported the images so they would become attachments. WordPress automatically copies all attachments onto the server, so we didn’t have to worry about getting all the images off our old server. Importing the images is just as easy. <wp:post_parent> is set to the ID of the post, <wp:status> is set to inherit and <wp:post_type> is set to attachment. The image path goes in <wp:attachment_url>, and the caption goes in <excerpt:encoded>.

We broke the XML files up by 1,000 posts at a time. All in all, we had more than 100 XML files. We also imported everything onto a local machine and then pushed the database back up to our webserver. All told, importing everything took several solid days of work.

If you’re working on a site much larger than ours, you might consider importing posts directly into WordPress using the API, but to be honest I’m not sure how much overhead that would save.

The script, which you’ll have to modify a bit but hopefully not too much, is on github.

A quick overview of our editorial workflow

Lauren Rabaino asked on a previous post for a video of our entire workflow. The whole process is actually pretty simple, so it wasn’t hard to record it.

Everything starts in Google Docs — that’s where the reporters write their stories, the AEs read them and the copy editors edit them. We interface with Google Docs via its API and WordPress via XML-RPC to move stories out of a folder and into the CMS. It requires a bit of cleanup, but for the most part everything goes smoothly.

All the stories are then saved on a local server as Indesign Tagged Text files, and prepared for print. Styles are applied, the byline and headline is added to the top of the story and a few other changes are made. We try to keep as much formatting from the web to print as possible, including bolding, italics, and a whole host of styles, particularly for sports.

We custom-built a plugin for InDesign that allows us to easily search WordPress and import the files from the server.

Video:

Credits

I wanted to credit a few people who were immensely helpful in getting the site up and running.

I’m William P. Davis, online editor at the Bangor Daily News. Todd Benoit is Director of News and New Media, and Martha Ward is Product Manager.

Mo Jangda, who now works for Automattic, which runs WordPress.com, wrote the Zoninator.

Juan Carlos Sanchez wrote the C plugin that integrates WordPress with InDesign.

Mark Jaquith, a lead dev for WordPress, and Ryan Duff provided much-needed freelance support.

We manage our servers in conjunction with Firehost. WPEngine is also a great host that we used for a time until we decided to go self-hosted.

Andrew Nacin, Daniel Bachhuber and Scott Bressler fielded a multitude of questions and developed excellent plugins.

Everyone, including Daniel, at CoPress who truly broke ground in this area.

And, especially, everyone who has ever contributed to WordPress core.

Open Source is superior because of the community around it.

Bangor Daily News completes final switch to WordPress

Wednesday, we pointed the last of our traffic to our WordPress servers.

We started planning the transition soon after I started at the BDN in July, and started beta testing the system in late August with our Sports section.

Stories are penned first in Google Docs, then brought over to WordPress via XML-RPC and pushed to InDesign via tagged text. It’s a unique system we built, for the most part, from the ground up, and we believe we’re the largest newspaper running entirely on WordPress.

Over the next few months we’ll be extensively sharing how we did it and open-sourcing much of the project. Our goal is to help other newspapers set up an easy-to-use, low-cost content management system. The setup is actually quite simple and easy to implement.

For the time being, feel free to leave comments with questions or e-mail me at wdavis@bangordailynews.com.

To get everyone started, I would recommend a few plugins that I think are must-haves for any news org on WordPress. The first, which the BDN commissioned from Mo Jangda, is The Zoninator, which allows you to order content by hand instead of chronologically.

Another is Edit Flow, which is an important tool for managing workflow through WordPress.

Scott Bressler‘s excellent Media Credit allows you to natively set the credit for images, instead of including the information in cutlines.

Co-Authors Plus, also by Mo, allows you to set multiple authors per post.

And CP Redirect is a good example plugin for how you might remap links from your old site. We used it as a template to avoid dropping links.

You might also wish to check out the Ben Franklin Project, from the Journal Register Company, CoPress, which, although not operating anymore, contains a trove of useful tips for converting, and a post I did in 2009 after converting my college newspaper to WordPress.

Chronicling the BDN on WordPress