Semantic search, Classification and Data migration: the winning team

Seven years ago, I had a mission to perform data migration from one system to another. One of the major challenges was to import parts inside a hierarchy of categories, which have been designed in the new system. Analyzing the legacy data, another hierarchy had been set, but this hierarchy had more than 900 entries, so users had mostly used a wrong category, making this information totally unreliable.

So we estimated we could try to use the description field of parts to classify the objects, guessing that the users had used meaningful words to describe their objects. The method had to be found.

So I imagined an algorithm to do so. The method was to analyze the words of the description, and to compare those words to a dictionary, providing as well a multiplication factor to each word depending on its position in the description. In parallel, I built the technical dictionary analyzing the description of roughly 500 000 parts, founding the most used words.

I shown that more than 75% of the parts could be automatically migrated using this algorithm. For the remaining 25%, I built an application which was providing the list of parts to classify, and the possible categories available in the new system, and we asked to experts to manually classify the remaining parts. Having done that, I enriched my dictionary with some new words that I had not been able to imagine the meaning (including some funny ones…). With the new dictionnary, we could be able to automatically classify more than 90% of the parts.

Then we set up an automatic procedure using this algorithm in order to migrate data at night from the legacy system to the new one, as both systems were decided to run in parallel for a given period of time. This system ran for one year, until all project data was migrated to the new system. Then the migration system was stopped, and put on archive. I created a semantic search engine without knowing it.

Years after, I have now to implement a search engine based on Exalead search engine. This technology implements semantic options, and hopefully I can reuse the dictionary I built seven years ago to provide more value this new technology.

My conclusion today is that there are several lessons I learnt from this experience:

  • semantic search can help migrate data
  • semantic search can help classify data
  • data migration activity can bring value for future activities
  • companies should pay attention building technical dictionaries, compiling words that users are using everyday
Found interesting? Share it!
  • email
  • RSS
  • LinkedIn
  • viadeo FR
  • del.icio.us
  • Facebook
  • Google Bookmarks
  • Technorati
  • Twitter
  • Ping.fm

Comments

13 Responses to “Semantic search, Classification and Data migration: the winning team”
  1. For over 15 many years Real Bookies has http://www.therxforum.com/showthread.php?t=918640&highlight= successfully assisted
    bookies from around the world to take bets from a large number of players on a pay for each
    head base.

    Our state-of-the-art bookie software program and the commitment from our clerks, lines people,
    administrators and companions have made our own price for each
    head sportsbook solution among the fastest expanding and most dependable bookmaking service provider names in the industry.

    Word of mouth may be the number cause new therxforum.com
    real estate agents sign up on RealBookies, which speaks
    towards the high quality associated with service provided to each agents plus their gamers.

    Upon signing up for the Realbookies family, you are going to receive your own personal assistant that will be right now there to solution any questions you may
    have concerning the company and the way to
    grow your business using our own pay per head sportsbook.

    Joining Realbookies will make your operation turn out to be par
    and level the particular playing therxforum field with the greatest online gaming companies in the
    world. The gambling software your players will be getting is the same by the
    biggest on the web sportsbooks today.

  2. Hi there, I want to subscribe for this weblog to get most up-to-date updates, therefore where can i do
    it please help out.

  3. It’s truly very difficult in this active life to
    listen news on Television, thus I only use the web for that reason, and take the hottest information.

  4. Great information. Lucky me I found your site by chance (stumbleupon).
    I’ve book marked it for later!

  5. This post presents clear idea in favor of the new viewers of
    blogging, that in fact how to do blogging.

  6. Hello, i read your blog from time to time and i own a similar one and i was just curious if you get a lot
    of spam feedback? If so how do you reduce it, any plugin or anything you
    can suggest? I get so much lately it’s driving
    me insane so any assistance is very much appreciated.

  7. It could be also great for those who have a lot more than one particular person within the house
    wanting a specific video game they are able to all obtain it just through
    a single download. With friv gaming, you can always find enough time to
    play your favorite games and even find more new favorites that you may be required
    to return and keep playing. Twelfth century soccer was a violent sport with little or no rules.

    Visit my weblog: dataskyall.com

  8. web says:

    Hey There. I discovered your blog the usage of msn. This
    is an extremely well written article. I will make sure to bookmark it and come back to learn more of your useful info.
    Thank you for the post. I will definitely return.

  9. reklamy says:

    There’s certainly a lot to find out about this topic.
    I really like all of the points you’ve made.

  10. Howdy! Would you mind if I share yor blog with my myspace group?
    There’s a lot of folks that I think would reallly enjoy your content.

    Please let me know. Thanks

    my web site commercial truck loans for 1st time buyers

  11. Darrel says:

    I pay a visit each day some web sites and blogs to
    read articles or reviews, except this weblog gives feature based posts.

    Look at my blog … equipment financing companies [Darrel]

  12. Undeniably believe that which you stated. Your favorite justification seemed to be on the net the simplest thing to be
    aware of. I say to you, I certainly get irked while people think about worries that they
    plainly don’t know about. You managed to hit the nail upon the top and defined out the whole thing without having side-effects
    , people could take a signal. Will probably be back to
    get more. Thanks

  13. Hi! I understand this is sort of off-topic however I needed to ask.
    Does running a well-established blog like yours take a large amount
    of work? I am brand new to blogging however I do write
    in my diary everyday. I’d like to start a blog so I will be
    able to share my own experience and feelings online. Please let me know if you
    have any kind of recommendations or tips for new aspiring blog owners.
    Thankyou!

Speak Your Mind

Tell us what you're thinking...
and oh, if you want a pic to show with your comment, go get a gravatar!

Spam Protection by WP-SpamFree