yDecode Forum

General Category => yDecode General Discussion => Topic started by: trikerbob on September 29, 2007, 01:12:04 AM

Title: How to get multipart files
Post by: trikerbob on September 29, 2007, 01:12:04 AM
I just purchased yDecode and I'm having trouble reading multipart post. Can some explain how to do it.
Thanks

Bob
Title: How to get multipart files or posts
Post by: yDecode on September 29, 2007, 02:16:19 PM
For details how to use specific options please consult help files but here is a brief tutorial of what is important when using multiparts. Please note that the multipart option is limited to non-yEnc files in the demo and also time-limited (no limitations at all in full version). Please follow the steps carefully. They are explained in detail with the reasons why are they requred - here but are very quick to follow and require just a few seconds to actually do.

NOTE: Initially this was a reply, but it grew to a small tutorial and it is being updated all the time. When it becomes too long we'll strip it into several sections. It may be a bit longer but it contains so much useful info so it is worth your attention if you use multiparts. You will learn a lot of useful things, we promise!

WHAT ARE MULTIPARTS AND HOW TO RECOGNIZE THEM

Movies and other large files on the newsgroups are split for various technical reasons into so called "multipart" posts. This is when first post contains only few Kb, next post another few Kb and so on until the last post. Let's say that the multipart file of 5 Mb has been split into 10 files that would make a post with 10 parts, 500 Kb each. This is a usual way of posting to the Usenet. The parts would be marked with counters like (01/10), (02/10), (03/10)... until (10/10) indicating which part exactly you're downloading. These counters are in the subject like of each part. There can be more counters and in that case other can represent file counters if more than one file was posted. In that case the counter might look like ... [1/4] ... (3/10) which would indicate roughly file 1 of 4 and part 3 of 10.

Please note that there are no standards regarding how to mark the file counters so there is some guesswork involved here but generally you'll be able to find the part 1 of some file relatively easy and this is all that is required for yDecode multipart function to work.

To view these, you need to "join" posts into a single file - joining each 500 Kb with the previous one etc. until you get original 5 Mb file. Of course, the file can be larger or smaller, this is just an example.

yDecode can help with this because it has automatic multipart decoder (it automatically joins the posts) but you need to learn first how to use it and there are some prerequisites for it to function effectively (for various technical reasons).

EXAMPLES OF SOME MULTIPART AND NON-MULTIPART FILES

You can recognize if a file is posted in multipart encoding or not by looking at its subject in most cases. Here are a few examples:

Quote
Non-multipart, JPEG image (subject line):
My picture set - Picture001.jpg

You should get single file called Picture001.jpg out of this one. This one is very easy and straightforward.

Quote
Multipart made out of 2 parts, JPEG image (two subject lines, two messages that need to be joined into one):
My picture set - Picture001.jpg (1/2)
My picture set - Picture001.jpg (2/2)

You should get a single file called Picture001.jpg which is posted out of two parts (1/2) and (2/2). As you can see (1/2) means part 1 of 2, and (2/2) means part 2 of 2. Usually these counters are in round brackets just like the above example.

Quote
3 different files, each one is multipart made out of 2 parts:
My programs [1 of 3] - Somefile.exe (1/2)
My programs [1 of 3] - Somefile.exe (2/2)
My programs [2 of 3] - SomeOtherFile.exe (1/2)
My programs [2 of 3] - SomeOtherFile.exe (2/2)
My programs [3 of 3] - Thirdfile.exe (1/2)
My programs [3 of 3] - Thirdfile.exe (2/2)

A variation on the above topic - also 3 different files, just a bit different numbering:
My programs [1/3] - Somefile.exe (1/2)
My programs [1/3] - Somefile.exe (2/2)
My programs [2/3] - SomeOtherFile.exe (1/2)
My programs [2/3] - SomeOtherFile.exe (2/2)
My programs [3/3] - Thirdfile.exe (1/2)
My programs [3/3] - Thirdfile.exe (2/2)

Note: square brackets usually indicate file X out Y and round brackets indicate part X out Y. This is a usual pattern for many posts on the Usenet. There is however an exception to that as well:
My programs - Somefile.exe [1/2]
My programs - Somefile.exe [2/2]

As you can see above - that example uses square brackets for counter. yDecode needs to guess from the above examples which is multipart and which one isn't so it is not perfect - but it works well as we've analyzed hundreds of posts and found some patterns that are implemented in yDecode recognition mechanism to make it as much reliable as possible.

You will get 3 files - Somefile.exe, SomeOtherFile.exe and Thirdfile.exe, each of them is made up of 2 parts.

Of course in real-life examples, the multiparts usually consist of tens, hundreds or even thousands of parts but you will be able to recognize the above pattern easily.

DIFFERENCES BETWEEN NON-DECODED YENC MESSAGE AND DECODED PART OF A MULTIPART FILE

Quote
An example (beginning text) of non-decoded yEnc formatted message:

=ybegin part=4 line=128 size=346200990 name=My Video.avi
=ypart begin=1536001 end=2048000
M$9@9I"###37@P:-$@C@O_83 SVSGB+H)@C(/3U.T=F:[KA#X_[P.^N#:%:X"....

If you get something like the above - the yEnc decoder is not working or you have incorrectly set the configuration of your news reader and/or yDecode. You have to fix the configuration. The presence of =ybegin means the message hasn't been decoded.

Quote
An example (beginning text) of decoded yEnc formatted message (but missing some previous part or for some reason not being displayed as "attachment"):

M%@C^K,IV4Q2:YU;N9$A/F3F%BX2)P**_]$51S$R;L`NH-N3T(U1XE"\<%FA\
MGYE?T%DBQ4"P&B7$YLD=6-RN(!.$B>&F<,Z1<$`!CU,H#0.7`.8P&),Q;N?*
MFMQUU=BMLXR-4\= PC6B*U5NK#8,JS]1-8!8S&L;V-<0>#X9Z#,YS?Y6MJ6M........


If you get something like this - that is decoded message (yEnc decoder is working and is configured well). The absence of =ybegin means the message has been decoded.

Why then if it has been decoded you still may see the above symbols?

a) yDecode is not set to automatically decode multipart messages (enable "get multipart as single" option)
b) yDecode is set to decode multipart messages but it you haven't retrieved enough headers from server (this is the most common reason)
c) Your server simply lacks the required headers and no matter how many of them you download some are missing and this is incomplete post that cannot be joined. Upgrading to better server may help (look at our links page)

HOW TO DECODE MULTIPARTS WITH YDECODE STEP BY STEP

STEP 1

Enable yDecode multipart option (check get multiparts as single in attachments tab). Do this before you want to use multiparts option. This is required first because yDecode needs to cache the data about multiparts which it recognizes when it downloads them. You can also choose how multipart option will behave if it considers the multipart to be incomplete. To learn more about this option please read the help file here: http://www.ydecode.com/help/manual_tab_attachments.htm

STEP 2

Now you need to make sure that you re-download the headers list (or the list of subjects if you will). This is required only the first time or when you want to begin using multiparts option. It is also required if you disabled the option and used this group for a while. Re-downloading the headers is required because yDecode needs to cache them and analyze them for multiparts. To do so - clear the cache of Outlook Express (look at troubleshooting to learn how: http://www.ydecode.com/help/manual_troubleshooting.htm#q003). Once you do this Outlook Express will be forced to re-download the list. All this is not required if you're loading a new group. Also it is not required in the future, because yDecode automatically handles all new incoming headers and automatically maintains its own cache.

Clearing the cache is essential so that the messages are re-downloaded through yDecode but only after the multipart option is enabled. If the multipart option was not enabled before, then yDecode didn't create its internal cache to recognize which posts are multiparts and which are not. So do this step the second after you enable the multiparts option as explained above.

Note: if you use some other newsreader except Outlook Express or Windows Mail which is very similar - you will need to find the appropriate option to re-download the list yourself but basically the procedure is the same. Most newsreaders probably cache the headers internally - therefore clearing the cache and re-downloading the list is essential.

STEP 3

Now, make sure you download as much headers as possible. Why is this required? Simply because the more of headers (or subject) you get, the more of them yDecode will analyze and recognize as multiparts or not.

Outlook Express and Windows Mail download only 300 headers every time you enter a group. This is usually not enough. You can increase this limit up to 1000. This means that each time you enter a group it will download additional 300 headers (or subjects). Or 1000 if you changed this number. But if you press an icon on the top bar marked Headers it will download additional 300 (or 1000) headers. Pressing this button a few times you will get a lot of headers. When you got them enough you may proceed. Unchecking this option will download all headers. This may be unpractical for groups containing many thousands of posts and headers icon retrieves only the latest ones first and the older ones on each sequential click so depending on your server and usage habits you will want to decide how you will use this - either by clicking Headers icon several times or by unchecking it to retrieve all headers.

Look at: http://www.ydecode.com/help/manual_troubleshooting.htm#q005 to learn more about how to tweak this option in Outlook Express and Windows Mail.

IMPORTANT NOTE: Sometimes it is better to uncheck the box to retrieve all headers but on some groups you can have millions of headers and that will seriously slow down your computer and your news reader. If that is the case, better increase the limit to 1000 (or some other reasonable number). Pressing Headers button in Outlook Express or Windows Maill will then get previous 1000 headers each time you press it. Usually, pressing the button a few times will retrieve enough headers for a few days - take a look at the earliest dates. To make sure that you have enough headers - use this simple tip:

If you want to download a message for example dated 5th Dec. - make sure that all the headers from 5th Dec. and also get all the headers from 4th Dec. as well. The idea is if you have at least a few headers from December, 4th - then all of the 5th have been retrieved as well and that you got all the required headers for 5th of December. Click Headers button enough times until you get all headers for a particular date. Then take a look at the list of subjects (headers) and see if the messages wrap from 5th to 4th - if they do - you're good to go with downloading a message from 5th of December.

Also, you can use the trick to "test" the header count with yDecode like this:

a) set incomplete multipart option to "do not download"
b) click "headers" button a few times
c) click on a desired message and avoid part #1 (001/123 or similar). Click on part 2,3 or any except part 1.
d) take a look what yDecode will report - there are 2 possible messages - "click on part #1 to download multipart" message and "multipart not downloaded" message
e) message #1 means: you have enough headers, yDecode got them all, recognized them and is good to go - click on part #1 to download message
f) message #2 means: there are not enough headers (click "headers" button a few more times), yDecode didn't got them or they are simply missing from server.
g) you need to avoid clicking on part #1 - until you get the #1 message when clicking on any part except #1 (2,3,4 etc...)

Why this works? Each time you click on a message - yDecode retests if it has got enough headers. Once it has enough - it will change the message it displays. So you can use this behavior to test if you got enough headers as well.

STEP 4

Having done all this now try to click on part 1 (e.g. 01/34 or 001/245) of the large file (like movie, mp3 or whatever). If the rest is available yDecode will be able to figure out this and download entire sequence automatically. If this is a large picture (usually split only in 2 parts) it will be displayed as normal picture in Outlook Express. If this is an MP3 file or something else, it will be displayed as paperclip attachment. If you click this attachment you can see how large it is - if it is larger than 500 Kb (from above example) then the download went fine and you can save this file and use it.

POSSIBLE PROBLEMS (AND SOLUTIONS TO THEM)


USING YDECODE TO OVERRIDE OUTLOOK EXPRESS, WINDOWS MAIL AND MOZILLA THUNDERBIRD BUGS

There are few bugs in Outlook Express, Windows Mail and Mozilla Thunderbird you should know about and which you can override using some yDecode options. The most significant one, related to multiparts is the behavior of all of these programs when saving very large attachments.

Typically if you download large attachment (300 or more megabytes) you won't be able to save it in Outlook Express (it will print out an out-of-memory error). Windows Mail may behave similarly but we have to test this out more. With Mozilla Thunderbird, it will not print out any error message but it will only detect and save the first part of the multipart (like, first 500 kB or so and not the entire file).

As most of the large files are encoded using the yEnc encoding - you can use yDecode to easily override these problems - simply enable the "Dump yEnc attachments to file". After that, download multipart normally and while it is being downloaded, yDecode will automatically save the file to the folder of your choice. Remember to occasionally clean up this folder from files you don't need, or move them to another folder. When a download completes, view the file from that folder directly. Don't do it before the download ends because the file won't be complete!

OTHER USEFUL TRICKS

*** How to learn if multipart is complete without counting all the parts manually

Click on part #2, #3 or any other than #1. Clicking on part #1 will download entire sequence but you won't be able to do this twice if you miss some headers (see problems above).

If yDecode reports something like "this is a part of multipart click on part 1" then it recognized it and clicking on part #1 will download entire post. If something else happens (different message - depending on state of the option if multipart is incomplete in attachments) this may mean you didn't download enough headers or headers are missing from the server in which case you cannot do much but find better server which holds more posts and has better completion rate or skip to another post.

Let's say you clicked on part #2 and you have if multipart is incomplete option set to "do not download". yDecode will report that it cannot download this message as the number of parts didn't match the proposed number of parts in the subject. You can now press headers icon several times to retrieve a few thousands of headers more. This will make sure that they all get through. Now try to click on part #3. Let's say it reports now that this is a part of multipart and you need to click on part 1 to download entire post. This means that now you've got all the parts and you can safely click on part 1 and yDecode will do the rest of the job. Many times you will not have to use this trick as all of the posts will already be there, but it could be a good habit to use it to test the multipart before you start downloading it. You can click on parts all the way down to the last part to test the multipart as long as it is not part #1 which will download entire post.

*** How download in the "background" in OE or Windows Mail

When using Outlook Express you will often want to browse other news posts while the large message is being downloaded. If you use only browser window then of course if you move away from the multipart message being downloaded at the moment which will stop the download. To avoid this - simply double-click the part #001 of the message. This will open it in a new window. Now, when the message downloads in the new window you can click back on Outlook Express or Windows Mail and browse other groups. Have in mind that this will require 2 connections on yDecode and that browsing will now be slower. We don't recommend you using background transfers like this on more than one large file even though it might work properly.