Webmaster Help | WebCrawler Issues (FAQs) How often the bot visit my page(s) ? -> The bot order might varies, it can take up a month to get on your site, or it might visit it daily depending of your domain's own rating. -> As a general rule: the robot visit each day one time any domain for a page, one page for picture(s) and one page for movie(s). (3x pages hit max per 24 hours period per domain) How to add my page from the listening ? -> Mail us at jabea@jabea.net. How to remove my page from the listening ? -> The bot respect ROBOTS.TXT. Please email me to get content removed fast in any case some material(s) got indexed that should not (but you need access to the ROBOTS.TXT). (jabea@jabea.net) --> In the root directory of the webserver, put ROBOTS.TXT ----> User-agent: * / User-agent: froggle ----> Disallow: /ressource/path --> See http://www.robotstxt.org/ for the exclusion standard.

Webmaster Help | Page(s) rating/rank (FAQs) -> The page rating is done on a lot of data collected, and statistic made from other page(s) (may sound obscure, but work good) -> The bot estimate the page use, thus making search more succesfull (ie, thumbnail page, technical site, forum, sale's site, etc..) --> Error can occur there for the estimate, but I offer that with no liability, and I don't intend to tell that my engine can be error free in that matter. (see my Terms of Service for that matter)

Webmaster Help | Bot Information (General Informations) Bot version : 0.09.5 - 2008-03-05 -> Follow HTML 1.0 standard. -> No rapid-fire. -> No obscure use for the data collected. -> "Bandwidth wise" (screenshot from a third party awstat page, found via a search engine) User-agent string : froGgle/0.0 BETA! (+http://www.jabea.net/bot.php) Current problem : -> Online database (the one of the webpage) are not sync'ed with my crawler database in a automatic way (not using clustered database right now due to host limitation). That make that the bot can visit your website, but the data will be online only when I will update the online database.

Webmaster Help | Bot Version History HISTORY.TXT: INSIGHT OF PROGRAMMING NOTE'S, WRITTED BY THE DEVELOPER (aka PHIL) 2008-03-05 ========== 0.09.5 -Main.c: Starting to add my WinMain to make it as a service. (I will make it easy to make it go back to a main for unix compatibility issue). -Main.c: Now: WS2_32.Lib libmysql.lib ole32.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib oleaut32.lib uuid.lib -Main.c: Was: WS2_32.Lib libmysql.lib ole32.lib 2008-03-03 ========== 0.09.4 -Main.c: Fixing my TAB/LF/CR (that I decommented), was not working ok, and I needed it for proper working. -Misc: Starting to think to make it as a win32 service.. 2008-03-02 ========== 0.09.3 -Main.c: Now stripping   & changing how I add some SP, and TAB/LF/CR (commented the code). :-) 2008-03-01 ========== 0.09.2 -Main.c: Databse sync now complete ! Work, but I got some errors, mySQL 4.1.x on host and 5.x.x on my machine. 2008-02-27 ========== 0.09.1 -Main.c: Using more mysql_init() and making a fucntion to replicate my database with my host database. 2008-02-17 ========== 0.09.0 -Main.c: Updating my stat init. -Misc: Updated my db-table.txt doc. 2008-02-17 ========== 0.08.9 -Main.c: Changed, and added more stat creation function. -Misc: Checking to update my online DB, changed my main query too to dig more site. 2008-02-13 ========== 0.08.8 -Misc: Testing, no longer crash on release/debug build ! (a small memory leak still there, but almost nothing) 2008-02-12 ========== 0.08.7 -Main.c: Changing my worker thread malloc. 2008-02-11 ========== 0.08.6 -Main.c: Problem seem a memory walk.. finding everywhere that I should had missed to end my string and such. 2008-02-07 ========== 0.08.5 -Main.c: Trying to find my Segfaul location from my release build.. Debug build work like a charm. -Main.c: From a thread call to strstr, now using mystristr, to allow better code checking. 2008-02-05 ========== 0.08.4 -Misc: Reading a lot about release build vs debug build & trying some optimisation in my program (thead stack size, etc..). 2008-02-04 ========== 0.08.3 -Main.c: Fixing a memory leak with my closing thread & adding some interface option ! 2008-02-03 ========== 0.08.2 -Misc: Testing - no crash, except if I use the release build.. 2008-02-02 ========== 0.08.1 -Main.c: On exiterr, closing mysql connection, some in sleep mode after some while running. -Misc: Added to my.ini ft_max_word_len=25 (431.9M to 310.2M - 28.177% saved! :-)) 2008-01-31 ========== 0.08.0 -Main.c: strncpy & strlen don't seem thread safe... fixed. Adding removal of mailto: link. 2008-01-30 ========== 0.07.9 -Main.c: Bug removed - need to make it bullet proof for directory. 2008-01-27 ========== 0.07.8 -Main.c: Thinking on how to fix my new bug (how absolute link are getting added te he :-))! 2008-01-20 ========== 0.07.7 -Misc: Find a bug in my mpg/jpg snap, abosulute link are not correctly added in the database. 2008-01-19 ========== 0.07.6 -Main.c: Fixed my img src check. 2008-01-16 ========== 0.07.5 -Main.c: Starting to check my image check routine, seem a bug in there. 2008-01-12 ========== 0.07.4 -Main.c: Adding some more insert for domain, updating my main sql query to be less intensive. 2008-01-07 ========== 0.07.3 -Main.c: Finishing on domain scan, added argv to my main console ! (n0w 1 f33l n3rd :p) 2008-01-06 ========== 0.07.2 -Main.c: Adding a scan to add the top level domain to prevent some hammering (like right now on blogsspot.com) -Misc: Reading over top level domain 2008-01-05 ========== 0.07.1 -Main.c: Dealing with some minor bug (the *.from data was not ok) -Misc: Now over 1,000,000 link added ! 2007-12-30 ========== 0.07.0 -Main.c: Creating a log file for error to allow better debugging ! :-) -Main.c: Access violation on ahref scan.. Starting to make it for multi-threading (about time!)... -Main.c: (GLINK no longer global tee-he! :) -see ver. 0.00.5) 2007-12-29 ========== 0.06.9 -Main.c: Removing a bug that cause the thread count to not decrement on exiterr. 2007-12-28 ========== 0.06.8 -Main.c: Changed exiterr() to display correctly mysql error. -Misc: Added set-variable=max_allowed_packet=16M in my.ini. 2007-12-06 ========== 0.06.7 -Main.c: Starting to add for main domain sorting. -Misc: Tested over two week, and crash almost never, last week not even one time. he he. 2007-11-27 ========== 0.06.6 -Main.c: Starting to change my pic grabbing for thread-safe function. 2007-11-25 ========== 0.06.5 -Main.c: Removed a bug with host: %sSP - rm -fR ! -Misc: Now thread-safe ! 2007-11-22 ========== 0.06.4 -Main.c: Testing my multithread. 2007-11-11 ========== 0.06.3 -Main.c: Debugging a segfault on mysql 'res', function not reentrant and thread-unsafe for winsock_.. & digdata(). 2007-11-10 ========== 0.06.2 -Main.c: Implented a global stack for the caller to handle thread data. 2007-11-09 ========== 0.06.1 -Misc: Thinking on how to pass my arg to my child thread to be safe.. 2007-11-08 ========== 0.06.0 -Misc: Thinking on how to pass my arg to my child thread to be safe. 2007-11-07 ========== 0.05.9 -Main.c: Changing to a connection to mySQL per thread, no multiple query per connect. -Main.c: (simplier code that way when not using mutex). -Misc: Starting to check my domain mail more often with outlook atm. 2007-11-06 ========== 0.05.8 -Main.c: Creating my heap for my thread. 2007-11-05 ========== 0.05.7 -Main.c: Testing someway to pass an unique arg to the new thread. -Misc: Reading a lot on multi-threading. 2007-11-04 ========== 0.05.6 -Main.c: Adding multithreading validation. 2007-10-30 ========== 0.05.5 -Main.c: Testing multithreading :-] -Main.c: Found & fixed my buffer overrun ! (was escaping oversize with mysqlrealescape.. > TITLEMAX) 2007-10-21 ========== 0.05.4 -Main.c: Fixing the title diging (no more using temporary http.txt to store data as from build 0.5.3). 2007-10-14 ========== 0.05.3 -Main.c: Global cCode & lSize no longer exist. 2007-10-08 ========== 0.05.2 -Main.c: Checking again my buffer overrun ! -Misc: Changed my table type in mySQL & bhougth a printer !, to debug my data offline 2007-09-16 ========== 0.05.1 -Main.c: Debugging my buffer overrun in digdata(). -Misc: Changed my temp location for mysql (my.cnf /tmp/ to /home/mysql-tmp). -Misc: (120m tmp filled by a 15mo table with a 8.8mo index.) 2007-09-16 ========== 0.05.1 -Main.c: Debugging my buffer overrun in digdata(). (title) 2007-09-12 ========== 0.05.0 -Main.c: Changed sprintf for snprintf. Changed HTTP/1.1 to /1.0 for all winsock functions. -Misc: Now using my tag to surf on the web. 2007-08-14 ========== 0.04.9 -Main.c: Removed some segfault, tested over 15 links, now 3165 page listed correctly ! -Misc: Found out the "Transfer-Encoding: chunked", read on that !! 2007-08-13 ========== 0.04.8 -Main.c: Debugging winsock_getbinary() for picture. -Main.c: Verifing some error control (reporting correctly the lastvisit, picture control). 2007-08-12 ========== 0.04.7 -Main.c: Verifing some error control (link size, etc..). 2007-08-06 ========== 0.04.6 -Main.c: Adding some error handling to all winsock_ function to not hammer some domain. 2007-08-05 ========== 0.04.5 -Main.c: Adding some stat to the db after some testing (domain that resolve but don't connect ie). 2007-08-01 ========== 0.04.4 -Main.c: Debugging live on some website, debugging some segfault. 2007-07-24 ========== 0.04.3 -Main.c: Starting some page stat ! (final .exe now over 100k ! :P) 2007-07-23 ========== 0.04.2 -Main.c: Cleandata finished, Cleandata_desc now working. -Php: Re-did the webpage layout to display the description. 2007-07-22 ========== 0.04.1 -Main.c: Cleandata almost done. 2007-07-18 ========== 0.04.0 -Main.c: Redo'ing my cleandata() ! And adding some check in my main() to not hammer same domain too much. 2007-07-17 ========== 0.03.9 -Misc: Finished a lot of research on _beginthreadex, and on threadless WinMain (to make a Dialog App, but from another thread to not lost my console). 2007-07-12 ========== 0.03.8 -Main.c: Now handling robots.txt correctly, and it affect already got data..!? (not sure when I did it ? (12/07 I think)). "ph1l's da sleePless" 2007-07-10 ========== 0.03.7 -Main.c: Finished to handle robots.txt on the fly! (only need now to check on new robots.txt to cancel already got data). 2007-07-09 ========== 0.03.6 -Main.c: Handling a bit bit bit more the robots.txt. 2007-07-08 ========== 0.03.5 -Main.c: Handling a bit bit more the robots.txt. 2007-07-07 ========== 0.03.4 -Main.c: Handling a bit more the robots.txt. 2007-07-05 ========== 0.03.3 -Main.c: Handling a bit the robots.txt (getting it now!). 2007-07-04 ========== 0.03.2 -Main.c: Changing last visit on domain differant for page/image/movie. -Main.c: Adding some code to handle robots.txt... but how? -bleh. 2007-07-03 ========== 0.03.1 -Php: Fixed a php var. bug (i != $i). -Php: Making the Website cooler, and adding the [Cached] option. 2007-07-02 ========== 0.03.0 -Main.c: Finding a Title lookup fault. -Php: Wrote the About, TOS and Bot Usage. 2007-07-01 ========== 0.02.9 -Main.c: Adding some statistic to the database for later use. -Main.c: Found a recvd another bug.. - added a select there for timeout (removed the break). -Main.c: Found a recvd bug - added a break in the while-loop. 2007-06-30 ========== 0.02.8 -Main.c: User-Agent: string now work. 2007-06-28 ========== 0.02.7 -Main.c: Working to be RFC compatible (HTML/1.1). 2007-06-27 ========== 0.02.7 -Main.c: Removed a possible memory problem in cleandata_href. -Main.c: Random check done for small domain. 2007-06-26 ========== 0.02.6 -Thumbs.php: Did some change for the thumbnail creation. (mysql vs mysqli) 2007-06-21 ========== 0.02.5 -Main.c: Started to add my random check on domain name. 2007-06-20 ========== 0.02.4 -Main.c: Optimized my break & download limit. 2007-06-19 ========== 0.02.3 -Main.c: Started to add some code for download/upload limit. 2007-06-18 ========== 0.02.2 -Misc: Checked to Sync Thumbnail, found php function ftp_* :) 2007-06-14 ========== 0.02.1 -Main.c: Added some sleep code ! - now less CPU intensive if no data to gather. 2007-06-13 ========== 0.02.0 -Main.c: Maked the program in a while(0), added a check to not check too much on the network. -Misc: Tried to find some info on how to get a list of domain registred.. will make a random thing. 2007-06-12 ========== 0.01.9 -Main.c: Neighborhood Fustration 0 - Phil 1 (for webpage) -Main.c: Added a status check on web page URI. -Main.c: URI with non-std char are now ignored to prevent database corrumption (yea, it happen te-hee) (33 to 126 0_o) -Misc: Did a bit of db change. 2007-06-11 ========== 0.01.8 -Misc: Did a lot of db change. 2007-06-07 ========== 0.01.7 -Main.c: Monitored my gethttp(), a error on a site, an antigather? -Misc: After a computer crash, reconfigured my computer. -Misc: Using VMWare server, did a test server on OpenBSD. -Misc: Reimported all the data in the db, now ready to continue coding! -Web.doc: Wrote all the step to configure my OBSD (Documentation) -Compileur.doc: Wrote my compiler config step (Documentation) 2007-05-17 ========== 0.01.6 -Main.c: Segfault 0 - Phil 1. 2007-05-16 ========== 0.01.5 -Main.c: Found a Segfault location ! (winsock_gethttp cCode) Corrected quickly ! (removed the code he-he) 2007-05-15 ========== 0.01.4 -Misc: Tested live on a thumb site how the grabbing go. (debugging win the tour) -Misc: Comment: Really slow(need multithreading). -Misc: Comment: Don't get all link(cgi link... cgi/?u... html://). -Misc: Comment: Some memory error in my database start to show up (need debugging). 2007-05-14 ========== 0.01.3 -Main.c: Maked my pic for movie scan work. (yea baby) -Main.c: Maked the ulong the standard over int. -Misc: All stack maked on function function to compile, changed the mysql.lib AND TESTED THE DEBUGGER, IT'S A CHARM! 2007-05-13 ========== 0.01.2 -Misc: Downloading VC Express: I hate to use MS product, but C:B debugger don't help me at all. -Main.c: Updated the code search (bug left corrected). -Main.c: Some change for VC (2x #define). 2007-05-12 ========== 0.01.1 -Main.c: Updated the code search. 2007-04-01 ========== 0.01.0 -Misc: Updated searchm.php, search.php, index.html & indexm.html : Added multiple page control. 2007-03-29 ========== 0.00.9 -Misc: Updated codeblock for debugging, debugged cleandata_link. 2007-03-08 ========== 0.00.8 -Main.c: Added function cleandata_linkend. 2007-03-07 ========== 0.00.7 -Main.c: Debugging the stack pile, changing code for the check (ulPos). -Win32/Ressource.rc: Added the file for version & copyright (win32 specific) (+some icons, muhaha) 2007-03-06 ========== 0.00.6 -Main.c: Starting the edit the a href check. -Main.c: Verified the pile stack (seem ok?). "pHil the null'ifier" 2007-02-28 ========== 0.00.5 -Main.c: Starting to add a pile stack for a href check validation (GLINK) -Misc: Updated 7zip to 4.44-beta to have AES-256 encryption in backup files! 2007-02-23 ========== 0.00.4 -Main.c: Changed %d -> %llu. (MY_ROW == unsigned long long != long) 2007-02-21 ========== 0.00.3 -Main.c: Added some new struct typedef. (movie thumbnail handling baby!) 2007-01-21 ========== 0.00.2 -Mysql: Changed movie struct. -Mysql: Changed rows len to be dynamics to save place. (67k to 3.4k on 18 pic rows) -Main.c: Started to add the .mpg snag. 2007-01-16 ========== 0.000.0.1 | 0.00.1 -Creation of the history file. -Main.c: Maked it that the console dump it's images directly in apache dir. -Thumbs.php: Changed thumbs.php to handle new path -Mysql: Added status to picture. Terms of service|About us|Bot information @2007 jabea.net