How to Scrape Linkedin Groups

Warning: Before I show you how to scrape Linkedin groups, you should know this..

It’s tricky.

Every tactic that used to work, no longer does.

Very few strategies worked to begin with. But Linkedin’s new UI change just broke the very few remaining strategies.

So I decided to use my technical skills to build a custom tool just for the community! It allows you to extract all the members of a group to load your pipeline and hopefully grow your revenue!

Here is a step by step guide on how to scrape Linkedin members from a Linkedin group that you are already a member of.

1. Gathering the member list

member-of-linkedin-group-screenshot

First, go to the Linkedin Group you want to scrape. You will need to be a member, but once you get there, click the Members.

Once you do, it will take you to the money page! Uh, I mean members page.

 

We want to extract these people, but in order to get all of them (or at least thousands), we have to continue scrolling in order to load more members on the page (as seen in image above).

But doing this manually will take forever. So here is a code snippet that you can use to automatically scroll to the bottom of the list:

//Code snippet Start
setInterval(function(){
window.scrollTo(0,$(document).height());
},2700)
//Code snippet end

To initiate autoscrolling, copy and paste this into your Chrome Browser Console like so:

linkedin-scrape-group-autoscroll

This could take a few minutes, so sit back and relax. Or go share this post with your friends

It will stop scrolling when you reach the end of the results.

2. Extracting in the users

 

Once we have all of the members loaded on our page, it’s time to extract them from this page.

To do this, go to the top of the members box and highlight “Members List” then Right click  > Inspect.

Once you do, highlight and copy (Copy Outer HTML) the line of code where it says item member-list-tab-layout-view (see below)

3. Extracting their linkedin URL’s

Now that we have copied that element, we can paste it into my custom script. If you don’t have it yet. Download it here.

This custom script will extract the real Linkedin URL’s and save them to a CSV for you.

In case you didn’t notice, Linkedin doesn’t actually expose their real url in groups.. They look like this:

https://www.linkedin.com/profile/view?id=ADcAAAAwfP4Be9WNSFRlR_R2vKUb3Xdlbo_mSx4m

But we need them to look like this

https://www.linkedin.com/in/dominickmalzone

Which is why we are going to programmatically find them.

Open the custom python script I shared with you and paste the element (that we copied from Step 2) into line 9 of the script where it says “Enter Group Code Here”.

(if you have a hard time, watch the video at the bottom of the page)

And then, run the script from your terminal or favorite text editor and let the code work its magic!

Once it’s complete you will have a nice list of all the names and Linkedin URL’s of every group member.

 

Download the code:
https://github.com/dominickmalzone/linkedinGroupScraper


Want to watch the video?

Want to bless your friends' feed?

11 thoughts on “How to Scrape Linkedin Groups”

  1. I’m assuming this works and I would like it to work and I’m trying hard to follow your instructions, but installation of Python and those additional libraries (libraries?) makes me sick. Would you consider enhancing your tutorial with installation steps?

    1. Yeah, I agree – it’s a little complex.

      I designed this tutorial for people who were semi-familiar with python, but there are links in the Github readme on how to install the necessary packages/libraries.

      If you have a specific point your stuck on, I’d be happy to help

  2. I was able to get everything set up right for python with beautifulsoup4, lxml, Selenium and chromedriver, but I’m a little iffy on what happens after you copy the linkedin code into python. Do you save the python with the code you insert into your spot and then run it in terminal? If so how do you run it in terminal as i keep getting a syntaxerror. Im sure Im doing something really simple wrong, but I’m just learning this stuff.

    1. I messed up the chromedriver part and I’m having trouble with that from what I can tell so far.

      I copied chromedriver into /usr/local/bin
      Put the path /usr/local/bin/chromedriver into line #9
      driver = webdriver.Chrome(executable_path=r’/usr/local/bin/chromedriver’)

      added member list html
      saved python as scrape.py
      opened terminal
      python3 scrape.py

      and this is what I get: /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python: can’t open file ‘scrape.py’: [Errno 2] No such file or directory

      Do you know what may have gone wrong?

  3. Ran into a bit of an issue- I ran everything and scraped everything, but when I opened the CSV file it was blank. Did I miss something?

    Thanks!

  4. Hello,
    “Once it’s complete you will have a nice list of all the names and Linkedin URL’s of every group member.”

    What then? Is there a way of pasting all the data on each URL to an excel file?

  5. Hey Dom,

    I am not sure that the scrolling script works with the new UI. Can you please check and let me know. Thanks

    1. Yeah it appears they have changed the members page. I will look into another solution but my gut tells me this new pagination will be much more difficult to do without being detected.

Leave a Reply

Your email address will not be published. Required fields are marked *