Intuition - Home of Wiki2Touch

You might want to know that there is another Offline Wikipedia solution. Have a look and Patrick’s site. Thanks Rizqi for the hint!

What I like about his application:

• The stripped down “database” files. Saves a lot of memory.
• Simple installation because of prepared indexes and modified download able data file (not sure if the later is allowed by Wikipedia).
• The way he’s doing searching. I really should add some AJAX stuff.

What I don’t like:

• How the articles are displayed. A lot of informations are simply not there because they are hidden by templates or tables.
• No language independence. You get English and that’s it.
  Patrick just dropped a note that you can use his software with other languages. More in the comments…
• To much bound to the OS/UI. I use my Mac to check new features and then I simply recompile it for the iPod Touch.

Joining both projects and then creating an easy to use indexer/data file creator for multiple platforms would be great.

Let me say thanks to everyone who’s actually involved or supporting the project in any way. Great support and way to go!

Timo, thanks for uploading the indexes and the rest to to rapid share. I appreciate it very much.

I’ve added the ability to let the user set the port for the server. Not released because neither the setting is stored nor retrieved from an eventually manually started server. Further I ran into a UI problem with 1.1.3. I don’t have 1.1.3 installed on my iPod touch, a friend of mine has on his iPhone.

While on 1.1.3: Looks like a lot of applications are not working anymore on that firmware. Mainly because the apps are started under the user account “mobile” and not “root”. Looks like that the mobile account is restricted. VT-100 terminal isn’t working anymore. Neither afpd.

For WikiServer it means that the server can be started from the WikiServer-App (and is running) but isn’t listing to the ip port. At least not 80. Maybe more luck with some higher port numbers (8080 or even more). Personally I will stick to 1.1.1 (or 1.1.2) until Apple releases the SDK by the end of the month.
Until then (on 1.1.3): You can start WikiServer using ssh as before and it works. But you can’t stop it with the application if started from the root account (using ssh).

Installation the software isn’t easy, I know. But here are some hints, annotations and tips as long as answers to the most questions:

• There is actually no convenient indexer. The indexing files are very large (en: 83 MB, de:17 MB). This is to much for my website. But it is possible to use Windows to index of course. Have a look at the comments.
• The web server “Apache” is not needed nor in any way involved. Apache cannot be used to server wikipedia pages. Of course you can use Apache for everything else, side by side.
• More worse, if Apache is installed, the wiki server cannot listen to port 80 and deliver pages. Port 80 is actually the default port for the wiki server and can only be change manually. Either change the server port for Apache or for the wiki server.
• To use a different port start the server from a command line (vt-100 terminal, ssh or so). Navigate to the “/Applications/WikiSrv.app/daemon” folder and run the server with “./wikisrvd -p 8080 -v-”. Instead of “8080″ you can use a different port if you like.
• I will add the ability to choose the port to the next release of the configuration program.
• The base path for the wiki server is “~/Media/Wikipedia/” (/private/var/root/Media/Wikipedia/).
• The wiki server can be used to deliver common html or other files. Simply put them into the base directory. This is then the root directory in the Safary browser. I.E. if you put “hello.html” into the folder ~/Media/Wikipedia/ you can load them in Safary with http://127.0.0.1/hello.html.
  Again: Apache is not used here!
• You can use Apache but then one of the servers (Apache or the WikiServer) must then run on a different port then 80.
• Everything that comes after the the virtual directory “wiki” (http://127.0.0.1/wiki/) will be threaded as the title of an articles and the software is looking for it. If found, the articles is displayed as a html page.

Hope it helps!

Not a perfect UI but usable:

I’ve just uploaded a new version. I think there are some great enhancements:

• Xapian is no longer needed. Indexing is a lot faster and the index itself is much smaller. Indexing the English edition took about 40 minutes, the German is prepared in less than 13. Still perl/grep is used so as soon as these are replace by a native C++ program I assume the English index is ready in less than half an hour.
• The basic server is now only one single file.
• The server starts as a daemon if you’re launching it with the “-v-” switch.
• There is a nice generic application for the iPod Touch/iPhone. It has a familiar icon and allows you to start/stop the server and launch the browser. Manually starting the server using ssh is not necessary anymore! (great if you need to reboot your toy)

I found out that many people were not able to install the software on their devices. The need to install Xapian was to difficult. So now, if you’re on a Mac, everything is ready in the new package. For the other OSs the source for the indexer is included.
Here is the new installation instruction (Mac OS X, see some annotations at the end):

1. Download the package here: WikiServer v0.20
2. Unzip the package (on the Mac this is done automatically and you will find a new folder in your downloads folder)
3. Download the Wikipedia archive files:
   en: http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
   de: http://download.wikimedia.org/dewiki/latest/dewiki-latest-pages-articles.xml.bz2
4. Place the downloaded archive file into the “Indexer” folder (inside the unzipped package)
5. Open a terminal, navigate to the “Indexer” folder and enter “make data”. For the German version please first edit the “Makefile” and replace “en” with “de” at the start of the file.
6. While the file is splitted and then the index is build copy the contents of the “Applications” and “private” folders from the “iPhone or iPod Touch” folder inside the package to you’re device. Use the provided directory structure below the “private” folder. Finally you should have the “WikiSrv.app” in your “Applications” folder and a folder name “Wikipedia” inside the “Media” directory.
7. The new application icon didn’t shows up automatically on the home screen. Either reboot you’re device (press and hold the home and on/off button some seconds) or install/uninstall an application using the “Installer.app”. Sending a “SIGHUP” to the “SummerBoard.app” works also. Any better ideas?
8. As soon as the index is done you have a folder called “en” (or “de”) inside the “Indexer” folder. Copy everything to the “~/MediaWikipedia/en” (or …/de) folder. Be sure not to delete the contents of the destination folder, especially for the German version the “language.config” file is needed. An also be sure not to end up with something like “Wikipedia/en/en/…” or so.
9. Start the “WikiSrv.app”, enable the server with the slider and launch you’re Safari.
10. Enter a search phrase. “test” is a good one for the first start.

As before: No warranty that you’re device is not crashing or so. Mine does it not but who knows?

Anyway: Have fun!

If you’re not using Mac OS X:
You need to compile the “indexarticles.cpp”. If your c++ tool chain (using gcc) is working from the command line simply enter “make indexarticles”. I will check if the source happily compiles using Visual C++ on Windows also. If not it should be easy to fix. No rocket science technology used. I promise. But probably you will get into trouble when it comes to use perl and grep. As soon as I find the time I will do a stand alone application for indexing.

I’ve found out that name ‘iKi’ is already in use by another application. So I need a new one for the Wikipedia offline reader.

Any suggestions?

I’m using ‘WikiServer’ in the meantime.

While writing an E-Mail to Yann I got the idea how to drop the need for Xpaian. Yann did a PHP Wikipedia Offline reader by himself but stopped the project after he found out that there is not bzip2 support in PHP on the iPod Touch/iPhone.

Searching using the new index will be limited to the complete case insensitive title. That’s enough for retrieving the templates. Because of a lot of redirects used by Wikipedia itself it didn’t hurt so much as it sounds.

Some more benefits:

• The index will be a lot smaller (app. 210 MB instead of 1.7 GB (english version))
• The 1.4 MB code file of the Xpaian library on the device is no longer needed
• The 0.8 MB additional libstc++ on the device is also not needed anymore
• As there is no need for any additional library anymore everything can be packet into one nice iKi.app file (including a nice icon); distribution using the installer is possible and very easy
• Creating the index is not needing Xapian anymore, can be done in one independent code file which is easy to compile or distribute in binary form; creating a nice GUI app is very simple.

So stay tuned, I’ve just started to implement it.

The content of this entrie is expired - here is the latest information about Wiki2Touch.

Topic says it all. A friend of mine with an iPhone, 1.1.2, has installed the Wikipedia Offline Player and it’s working nice.

I’ve also created a nice icon for the home screen using Erica Saduns “OpenURL”. As soon as Apple releases the SDK for the iPhone/iPod Touch the installation will be much easier. Hopefully it will be in February this year as the promised.

First: Today I had found some time to iron out some nasty bugs:

• The handling of the “language.config” files is now much improved. For the German version (or other languages of course) the correct month and day names are now also stored in that file.
• Templates in template name are resolved: That make the “Main_Page” working. Including “Today’s featured article” and “On this day…”. Of course, the first is not changing so much.
• Minor issues: The app is now compiled with -O3 optimization. Upon receiving a SIGTERM the socket is properly closed.

Unfortunately I was not able to set up a project on SourceForge or Google codes today. Only a lack of time. But if you like you can get the application and check it out by yourself. The link is below. But be warned: This is not easy.

There is not much documentation but here are some guidelines:

• Read Thanassis Tsiodras article “Building a (fast) Wikipedia offline reader”
• Don’t use his “quickstartindex.cc” but mine. You will find it in the folder “Changed quickstartindex”. Using my one will decode URL encoded titles. “Black%20&%20White” will become “Black & White” and is then indexed under this title.
• Put everything in the “iPhone or iPod Touch (1.1.1) folder onto your device. I’ve added the proper directory structure so you should know where everything has to go.
• Put the produced index and the splitted bz2-files to the proper folder on you’re device. Have a look into the zip and you know the right location.
• English should work but I never put the 4.5 GB to my iPod touch. But I’ve checked it on my Mac. (BTW: 2.9 GB is the size of the bzip2 files, 1.5 GB is the size of the Xapian index).
• If you like to use a different language: Create a folder with the language code as its name and copy the “language.config” from the German installation and edit it.
• The application automatically detects which languages are installed. Every subfolder is scanned for a “language.config” file. But the main “index.html” is not aware of that. Change this file to the proper language code. Actually it is set to “de:” for the German version.
• Start the wikisrv on you’re device using ssh. I prefer the port 80 on the device but 8080 on my Mac. 8080 is currently default. So I use “wikisrv -p 80 -v- &” to start the server on port 80. The -v- switch suppresses some info messages.

Some warnings:

• I’m not responsible for any damage to you’re device or so. Mine is still working and it never needed a hard reset because of using the Wikipedia offline reader.
• You might have found out that some knowledge about Darwin/Unix and a jailbroken device is necessary. So forget about using iTunes for copying or so. Use MacFuse, Cyberduck or something else. PuTTY, WS_FTP or other PC applications are fine, too.
• Beware: This is beta and you will see some minor rendering issues. But most of the articles are displayed fine.

As soon as Apple has released the developer kit of the iPod touch/iPhone I will switch to this.

You can download the current version here: iKi_20080106.zip.

Have fun!