The Wonderland Board

LexieTheFox wrote:

VirtLands wrote:The following attachment is a sorted list of member names, ID's, emails,
compiled from 241 member webpages. Click on the attached download.

Enjoy.

Ok um... I'm not ok with my ID or Email being given out on that mirror website. Please, when its up. Display my name ONLY.

Agreed Except only showing my username.

LexieTheFox wrote:

VirtLands wrote:The following attachment is a sorted list of member names, ID's, emails,
compiled from 241 member webpages. Click on the attached download.

Enjoy.

Ok um... I'm not ok with my ID or Email being given out on that mirror website. Please, when its up. Display my name ONLY.

Firstly, the informations exposed by VirtLands has nothing to do with me. Secondly, if you choose to publicly display your email address the bot will grab them, but they will never be shown in the public. If you did not, the bot will not attempt to grab them.

(For anybody that is worried about the exposure of emails in the Off-Topic area - the bot is never configured to access the Off-Topic area in any way, nor is there code to scrap information only accessible by a moderator)

I don't want anything but my username being given away either, please.

Guys, your ID/Email/other personal stuff isn't going to be given away.

Wonderman109 wrote:I don't want anything but my username being given away either, please.

If you did not give away your personal information publicly, the bot won't be able to get them.

I have removed the PCPuzzle Member's List.TXT attachment from page 1.

Wonderman109 wrote" Ị̤̣ d̤̣̣̣̣o̤̤̣̤n̤̤̤̤̤̣'̤̣̣̤̣̣ṭ̣̣̤̣̤ ẉ̤̤̤̤̣ạ̣̤n̤̣̣̣t̤̤̣̤̤ ạ̤̣̣̤ṇ̣̤̣y̤̤̣̣t̤̣̣h̤̣̤i̤̤̤̤ṇ̤̣̣̤g̣̣̤̤̤̣ b̤̣̤̤̣̤ṳ̤̣̤̣̤t̤̤̤̤̣̤ m̤̣̣̣y̤̣̤ ṳ̤̣̣s̤̤̤̣ẹ̤̣̣̣ṛ̣̣̤̣n̤̤̣̣̣̣ạ̣̤̤̤̤m̤̤̣e̤̤̣ ḅ̤̣̣̤ẹ̣̣̤̤ị̣̤̣ṇ̣̤̤̤g̤̤̤̣̤̤ g̤̤̤̣i̤̣̣ṿ̣̤ẹ̣̣n̤̣̤̤̤ a̤̤̣̤̣ẉ̤̣ạ̣̣̤y̤̤̣̤̣̣ ẹ̣̤ị̣̣t̤̤̣̣ḥ̣̣̤ẹ̤̤̤r̤̤̤̣̣,̤̣̣̣̤̤ p̣̤̣̣̤̣l̤̤̤̣ẹ̣̣̤̣̤a̤̤̣̣̣s̤̤̤̣̤e̤̤̤̤̤̤.̣̣̣

That is what I mean by "public information".

EDIT: I am happy to announce that the spider is finished. Now I am testing the spider then moving on to the Pipeline (the medium that stores things into database). The trickiest part of the Pipeline would be the html2bbcode function, which turns raw HTML back to BBcode.

Debugging took more time than I thought it would. It seems like I understood some of the concepts wrongly, so now I have to redo some bits of scraping.

(And as part of the test, I now have almost everything shown to the public)

Nothing much has happened over the time, but the scraping bot is coming into shape. Chris (christ) has offered to work together for the website, his experience in PHP scraping will definitely come in handy.

I now have school and have to apply to university (eek), so I won't be able to work on this as much as I could in the summer holidays. Hopefully I have the willpower to keep this project alive...

I'm aiming for MIT, and you know how people there like to work day after night after day.

So I'm not sure.

Development has halted because I had a bit of school work and unexpected projects coming in.

It will restart in... *insert restart time here*

Because the development isn't going anywhere, I'm pushing the code on GitHub for everybody to see. Hopefully I'll have more time in Christmas break.

Note: This code is broken.

https://github.com/tyteen4a03/wlf_scrapy

tyteen4a03 wrote:...Note: This code is broken....

So, is it broken as in incomplete, or broken as in shattered ?
---------------------------------------------------

Well, this part was interesting anyway:
sample code from TyTeen's items.py:

class User(Item):
"""
A user.
"""
userID = Field()
username = Field()
joinDate = Field()
totalPosts = Field()
avatarName = Field()
location = Field()
website = Field()
occupation = Field()
interests = Field()
email = Field()
msn = Field()
aim = Field()
yahoo = Field()
icq = Field()
signature = Field()

Rest in peace, code...

Incomplete. Sorry if I wasn't clear enough

A major reason why it's incomplete comes from pipeline.py. I haven't finished the htmlToBBCode method yet (and it's hacky, I tell ya). Otherwise, the code runs (with a lot of exceptions yet to be fixed)

So is this still a thing, Tyteen?

How's everything been doing the past few months?

Technos72 wrote:My answer to good idea:

AGH TOO MUCH PONY AHHHH

jdl wrote:So is this still a thing, Tyteen?

How's everything been doing the past few months?

Nothing has really happened in the past few months, I had freelance projects writing XenForo addons, exams and a very big video project (think full-length movies). Now I'm finishing a music video that is very important to me, then I'm moving to Denmark for a year of foreign exchange.

My coding skills has grew a lot since I started developing XenForo addons, and I can see the new Wonderland Archive being based on XenForo's Resource Manager addon. It will (as always) be a lot, lot more easier if Patrick would switch to XenForo and give me direct access to the database for post conversion, but it's not happening soon, if ever.

This project will be out of hiatus somewhat soon, but this project is low on my priority list - I need to start coding for profit so I can make a living. As for when the hiatus will be over, I honestly don't know. Feel free to take the current WLF Scraping code and expand it into a full bot and scrap the forum - with data, this will all be a piece of cake.

tyteen4a03 wrote:a very big video project (think full-length movies

*twitch*

As a wannabe filmmaker (who doesn't have a camera nor the money for it, nor have I written a finished script) I'm highly interested in knowing what the project is and what your role in it was.

Muzozavr wrote:

tyteen4a03 wrote:a very big video project (think full-length movies

*twitch*

As a wannabe filmmaker (who doesn't have a camera nor the money for it, nor have I written a finished script) I'm highly interested in knowing what the project is and what your role in it was.

Everything. It's not even a movie movie - just a lot of photos plus heaps of funny content of my 100 friends at school put together in a Ponies - The Anthology way. It seems to be a success - my friends enjoyed it a lot.

However, my Music Video coming up will be a real professional production - stay tuned.

A lot has happened in the past 48 hours and I'm sad to announce the MV will need to be postponed at least a year later. However, this will mean I will have a little bit more free time working on this.

I've shot MS an offer of my spare XenForo license - now patiently waiting for a response. If MS gives switching to XenForo a green light, I'll bump the priority of this project, and incorporate an API for future Wonderland games to directly upload levels and adventures. If not, I'll continue my work on the scraping bot - it's quite near completion as I plan on not scraping some very outdated data (like IM screen names - who uses any of the listed IMs in the Profile anymore?) and slimming down the functionality (like built-in html to BBcode conversion - these HTML mess I can fix by hand).