New version 0.46 is out
As promised I’ve uploaded the newest version. Download it at the google code page.Whats new?
- A new indexer makes it a lot more easier to use. Simply download the desired articles bundle from Wikipedia and start the indexer with “indexer [name-of-the-downloaded-bundle]”. After a while (English app. 50, German 12 Italian 6 minutes) there is only one file called “articles.bin”. Put that on your device and your done.
- The indexer is available as an executable file (
Windows version is coming soonis included). No development tool or experience necessary anymore. - A build-in Wiki database which displays error messages a lot nicer and also has a help page.
- You can turn of the feature that templates will be expanded. Sure, a lot of info is lost then (for example the nice period system on the articles about an element) but it is much faster. The decision is up to you.
- The port can easily changed and is now 8080 by default. This prevents problems with Apache (if installed but that is NOT necessary for Wiki2Touch).
- The home page has a “Help” button.
- Cleaned up “Media/Wikipedia” folder. Only the folder for the installed database files are left. Everything else…
- … is gone to the “Wiki2Touch.app” bundle. That makes installation much easier.
- Wiki2Touchis now “Open Source”, based on the GNU license V3. Source is freely available.
Certain rendering issues will be addressed in the next version. Thanks to Nicolas (from iphone-ticker) for running the forum. Also thanks to Kim and Uwe for starting to help. Before I forget: Uwe is working on an installer package. But he’s also busy IRL.
Februar 17th, 2008 at 17:48
Thanks for the update! I had noticed that the zh language.config had some incorrect text strings. I want to know where these strings will be used, are they for the display only or will they be used in the engine to resolve for example the template pages. If they are for display only I believe it is better to use the default English ones.
Februar 17th, 2008 at 18:27
Sanford-
you’re welcome. The reason why the app need this “language.config” stuff is, that this is not defined inside the articles.xml.bz2. Or I’m to stupid to find them.
Here is a short list and for what they are needed:
Template= Very importend if templates are used. This prefix will be added to the template name and then is the template retrieved from the database. I.E. {{Test}} will become “Template:Test” in the English edition. Then the “article” is search with that name. So if the prefix is wrong the template isn’t found (red text Template:Test, the prefix for the error message is always “Template”).
The rest is easy: Title of the “Table of contents” box at the start of the page, name of the “Categories” label at the end of the page. I’ve “stolen” these characters from the chinese wikipedia). Then then name of the days, month. You will see them when you navigation to the “home” page. For chinese this is http://zh.wikipedia.org/wiki/首页. The home page is working.
Februar 17th, 2008 at 18:29
Sanford-
if you’ve changed it would you send me back the new “language.config”. I will place it to the bundle.
Ahh, one more thing: To find out what the prefix is for template is not easy. Maybe in the help section somewhere. Hardy not readable for me. For Italian and Spanish I guessed right, for French I think not.
Thanks!
-Tom
Februar 18th, 2008 at 03:55
Hey Tom,
how can I upgrade to 0.46? How exactly should I proceed? Thank you so much!
Februar 18th, 2008 at 17:40
The previous indexer make happily on my linux box, but with the same set up I could not get it to make under the same configuration… Would you by any chance know why?
gcc -c -O3 indexer.cpp -o indexer.o
indexer.cpp:66: error: conversion from ‘int’ to non-scalar type ‘_G_fpos_t’ requested
indexer.cpp:86: warning: #pragma pack (pop) encountered without matching #pragma pack (push)
indexer.cpp: In function ‘int index(int, char**)’:
indexer.cpp:659: error: no match for ‘operator*’ in ‘1.0e+2 * currentPos’
indexer.cpp: In function ‘int get(int, char**)’:
indexer.cpp:906: error: no match for ‘operator+’ in ‘fileHeader.FILEHEADER::indexPos + (((long unsigned int)index) * 4ul)’
indexer.cpp:911: error: no match for ‘operator+’ in ‘fileHeader.FILEHEADER::titlesPos + titlePos’
indexer.cpp:941: error: no match for ‘operator+’ in ‘fileHeader.FILEHEADER::indexPos + (((long unsigned int)foundAt) * 4ul)’
indexer.cpp:943: error: no match for ‘operator+’ in ‘fileHeader.FILEHEADER::titlesPos + titlePos’
indexer.cpp:954: error: cannot convert ‘fpos_t’ to ‘__off_t’ for argument ‘2’ to ‘int fseeko(FILE*, __off_t, int)’
Februar 18th, 2008 at 18:30
Sanford-
looks like that you’re running into problems because of “long” instead of “long long” varibales. Because the filesize can be more than 2GB (i.e. the english edition is that large) we need 8 byte integers. Both, titlesPos and indexPos are 8 byte unsigned integers, not 4. So check how large the type “fpos_t” is for you. Must be 8, not 4.
In line 66 you can skip the #pargma pack(). I wonder why it doesn’t makes problems on the MAC GNU gcc compiler.
-Tom
Februar 19th, 2008 at 02:22
A google search reveals that fpos_t is a complex structure in linux and should not be accessed directly as a base type object. I can try to fix the problem when I am more free.. gotta go..
Februar 19th, 2008 at 10:57
Sanford-
I’ve not prepared the windows version of the indexer/getter. I’ve removed the warning with the “#pragma pack(pop)”. Then I had to a #define for fseeko, which is not known under windows. I’ve used _fseeki64 instead. After the changes VS7 was happy to compile it.
That indexer is wokring fine using Windows, I’ve just checked it against the chinese database. Works fine!
Maybe it helps if you simply #define fpos_t as “long long”. It turn out, that support for large files is handle different on any system. To bad.
Februar 19th, 2008 at 10:57
BTW:
The windows indexer and the changed indexer.cpp sourcefiles are already up. I’ve also added a necessary header to the “dev-libs”-package.
-Tom
Februar 19th, 2008 at 18:18
zh language.config:
# this is the name of the prfix which is used before templates, don’t forget the trailing colon
# default is Template:
templatePrefix=Template:
# which character is used as a decimal seperator (us=, de=.); the opposite seperator for
# the fractional part is then handle automatically; only the first character is used (even if it is
# a space character)
# default is ‘,’
decimalSeperator=,
# the title of the table of contents (TOC)
# default is “Contents”
tocTitle=目錄
# the name for the list of categories
# default is ‘Categories’
categoriesName=分類:
# names for the days
# default are the english name
Sunday=日
Monday=一
Tuesday=二
Wednesday=三
Thursday=四
Friday=五
Saturday=六
# abbreviated names for the month
# default are the english ones
Jan=1月
Feb=2月
Mar=3月
Apr=4月
May=5月
Jun=6月
Jul=7月
Aug=8月
Sep=9月
Oct=10月
Nov=11月
Dec=12月
# names of the month
# default are the english names
January=1月
February=2月
March=3月
April=4月
May=5月
June=6月
July=7月
August=8月
September=9月
October=10月
November=11月
December=12月
Februar 20th, 2008 at 14:35
Hi Tom!
I’m still impressed by your excellent work. Packing everything in one single file is a brilliant idea btw. Handling many files was not so much an issue for the dumps but it became a clear issue when I first started supporting images.
When I released iPhonePedia I didn’t mean to start a competition…
I’m actually willing to contribute on three features that are very important to me and available in iPhonePedia which I will continue to support as long as they’re not in Wiki2Touch:
- support for images (probably no one will want it for the english version because of the size, but clearly it’s wonderful to have that)
- support for math. It may not be fundamental to everyone, but it’s so frustrating to not have when needed.
- a full text index. Really man, you need that. I don’t quite understand whether you have it or not, but clearly when I search for “rolling stones” I WANT to have articles named “the rolling stones”
The final refinement would be a portable indexer, but in fact I think the real solution to that is to have a reliable release process for the data files so that no one should have to generate them.
On my side all languages are available (my indexer has autodownload so it’s only one command line per language) with optional math and image packs now (took some time for all the image downloads!), but I simply don’t have the hosting space…
Februar 20th, 2008 at 16:23
Yann-
thanks for the message. No, I don’t want to start a competition either. Working in a creative team is better than alone.
And you’re “da man” today because you got me to the idea: Download the stuff directly from Wikipedia and inexing it on the fly ON THE IPHONE!
I will try out how fast that is. Indexing the 1 GB German Wikipedia takes app. 15 minutes on my Mac. I really want to know how long that takes on the iPhone.
Images are nice. I’ve checked the image pack which was running around here (for the english wikipedia) and it worked nice. You’re right with math symbols, but that has to wait. Only a few articles are using them. And for the full text search: Yes, having one is great. But I’ve not missed it until now. Why? Well, let’s take your example: “Rolling Stones” is redirected to “The Rolling Stones”. Wikipedia itself didn’t has a full text indexer AFAIK. Our name it Google
-Tom
Februar 20th, 2008 at 19:09
Yann-
I had a quick look on your site. Your list of screenshots is amazing. Maybe I should compile one myself, too. Multi-Language and UTF-8 support was in all the time.
Really bad that we can’t merge the advantages of both readers. I really like your indexer which downloads the images automatically. Something I should consider to add also. The images are looking very nice. Hey maybe you can pack all the images together and send them. Should work with Wiki2Touch out of the box.
But, sorry to say, to install your solution you must have a diploma in computer science. A common, even interested windows user isn’t able to run Perl or PHP on his box. Forget about TeX.
So actually I’m focusing on the matter to make the installation as user friendly as possible. And if it’s really working that the indexer can download and run on the device in an affordable time that would be it. Download the complete app with the installer (which is possible today) and configure the rest in a user friendly GUI.
BTW: I’ve now added an Ajax driven search front-end. Quite nice. After you’ve typed four letters you get a list of the first 25 hits which you can directly click. Due to binary search that is fast as hell.
-Tom
Februar 20th, 2008 at 19:13
Sanford-
thanks for the chinese language.config. I’ve already added it. Maybe a good start to learn Chinese
-Tom
Februar 20th, 2008 at 19:33
The download shouldn’t be a problem since curl was ported to the iphone and will have all the error management you’ll need for large downloads. I would go for that. But my experience with the indexing from the itouch (my initial indexer was in PHP so I could run it from the itouch) was not that good. With a compiled program you may be more lucky though, in which case that will be the key to opening the solution to non-techies.
You have a good point on the rolling stones, but I can find dozens of other (totally imaginary but credible) examples that are not only about removing ‘the’ or ‘a’:
Typing ‘new york hotels’ to find ‘list of hotels in new york’
Typing ’sorting algorithms’ to find ‘comparison of popular sorting algorithms’
Typing ‘movies 2007′ to find ‘the hundred most popular movies in year 2007′
Typing ‘kerry bush’ to find ‘why did George Bush lose over John Kerry’
The first letters of an article are not always the most relevant…
I’m not surprised you didn’t miss it since as long as you don’t have this, you don’t know you need it because you just don’t see how many articles you miss and which potentially end up being completely inaccessible…
Who knew they needed Google when all we had was Altavista?
Februar 20th, 2008 at 19:53
Hey Tom
No pb to send you all the images, so you can give it a try, which I really encourage you to do.
They will be available on TPB anyway.
I concatenate them in up to 256 binary files per language to avoid having too many of those to copy. Same for math.
I’m happy with the size it takes but it’s true the English version is simply too big for most people.
The indexer doesn’t need PHP, and installing activestate is not any harder than installing Acrobat. Frankly it’s nothing compared to having a compiler installed!
LaTeX is another story but the popular windows distro is also just a wizard installer…
And most users probably won’t need it anyway.
Actually most users should just download the files and not compile them
Why don’t we just create an entry in wikipedia to compare our solutions in a nice table, to help people make their choice?
We’ll just both reference it from our websites.
Februar 21st, 2008 at 17:21
hi all
first of all: great work. i used the app on my 1.1.2 iphone a couple of days and it was great!
i updated now to 1.1.3 and the app is no longer running. on the google code page you state that it can be installed on 1.1.3 but with some additional steps. could you outline the steps necessary please?
thanx in advance
teefix
Februar 21st, 2008 at 17:42
Teefixy-
Have a look into the comments section of the article “Thanks guys!” Or simply search for 1.1.3 in this blog.
Nicolas describes a method to let the app run under 1.1.3 in the first two comments of the list. The disadvantage doesn’t exists because of 1.1.3, but because of the fact that if you bind an application to a port, unbind it the os wait some second before it let you bind again to the same port.
I meanwhile found a fix for it. It’s already in v0.5 which is nearly ready and I will release the version soon.
-Tom
Februar 21st, 2008 at 22:11
first of all i want to say this is really a great app, but i have a problem with getting the installer to run with windows XP, when i double click the installer it starts and immediately shuts down. so how am i able to use the ./index… command? please help, thanks in advance ulfilus
Februar 21st, 2008 at 22:29
i meant the indexer
Februar 21st, 2008 at 23:46
ulfilus: don’t double click it to run it. use start -> run. then enter the command there.
Februar 21st, 2008 at 23:56
thnks John I tried that, but can you give me the exact command? because if i type ./indexer xxwiki-…xml.bz2 (with de instead of xx of course) it says file cant be found but if i type in the path where the indexer is located it starts again and then shuts down
Februar 22nd, 2008 at 18:19
Actually, click start -> run,
Then type: cmd
This will open the command prompt.
Then use the DOS commands to get to the file path.
Then once you get in the correct path, type indexer to run the program, and it will give you the list of options you can type in to get the indexer to work.
Hope that helps!