Spotify’s Music Recommendations Lambda Architecture

  1. 1.
    Spotify’s Music
    Recommendations
    Lambda Architecture
    Esh Kumar @eshvk
    Emily Samuels @emilymsa

  2. 2.

    Overview
    ‣ Why Lambda?
    ‣ Use Case: Discover Recommendations
    • Batch Architecture
    • Real-time Architecture
    • Challenges
    ‣ Future Work

  3. 3.

    Why Lambda?
    • 1 new user every 3 seconds.
    • Contextual, time based recs
    more & more important

  4. 4.

    Discover
    Recs

  5. 5.

    The Discover Page
    Algorithmically generated fresh
    recs for users.

  6. 6.

    The Discover Batch Pipeline

  7. 7.

    Machine Learning Deep Dive

  8. 8.

    Word2Vec
    Words with similar
    contexts have similar
    meaning

  9. 9.

    Word2Vec
    King – Man + Woman = Queen

  10. 10.

    Annoy
    • Approximate
    Nearest Neighbors
    Oh Yeah!
    • https://github.com/s
    potify/annoy

  11. 11.

    Batch Architecture
    Strengths

  12. 12.

    Intro to
    Storm

  13. 13.

    Storm
    • Distributed real-time
    computation system

  14. 14.

    Storm @
    Spotify

  15. 15.

    Real-time Architecture

  16. 16.

    • Workers die -> Cascading JVM
    Process death
    • Memcache flakiness
    • Cassandra JVM problems due to
    write/overwrite pattern
    Challenges

  17. 17.

    Future/Ongoing Work
    • Simplify the topology
    • Keep listens for 24 hours
    • Ongoing work on other
    real time personalization
    features.

  18. 18.

    Questions
    Esh Kumar eshvk@spotify.com
    Emily Samuels esamuels@spotify.com

[tribe_event_inline id=”3760″]

Content related to: {title:linked}

[/tribe_event_inline]