Discord, Elastic Stack for Threat Intelligence

Introduction

This has nothing to do with threat intelligence about Discord or Elastic Stack, but rather leveraging these platforms as a Cyber Threat Intelligence (CTI) platform.

Overview

I was looking for new ways to bundle up RSS feeds, Twitter feeds, and other sources that make up the never ending wave of cybersecurity news, IOC’s, and information. I was running Tiny TinyRSS in a Docker container and cramming as many useful RSS feeds I could find into it as well as creating some filters for relevant information.

Since it’s simply and RSS reader, there’s obviously no type of Twitter integration, so I was also running this side-by-side with TweetDeck, configured to show the feeds of the Twitter accounts that I found to be most relevant in gathering and monitoring for CTI. I also couldn’t do any “transforms” on the RSS data or send to Discord for that matter.

In this post, I’ll discuss:

• Why Discord for RSS, Tweets?
• Why Elastic Stack (ELK) for RSS, Tweets?
• Limitations with Discord “bots” and hosted services
• Self-hosting with Huginn & sending RSS feeds to Discord
• Sending Tweets to Discord with Discord-Twitter-Bot
• Ingesting RSS feeds, Tweets into an Elastic Stack

Why Discord?

I just wanted a way to get this into a single platform that could leverage automations, API’s, Webhooks, etc. to get all the information I wanted in a single place. I know it seems like an odd choice, and maybe impractical, to use Discord for such a platform, but this blog post I stumbled across inspired me to take it a step further and also find new solutions to some of the limitations the Discord Bot services used provided.

Why Elastic Stack (ELK Stack)?

Ever look at the “About Me” page? Dashboards! Seriously, though, there are some limitations to how practical sending Tweets, for example, would be to Discord. Especially if I’m looking to extract and parse things such as file hashes, IP addresses, etc.

Leveraging Logstash within the Elastic Stack can make this a relatively easy task, especially since there’s a Twitter input plugin for Logstash already available.

Oh, and there’s a Logstash RSS input plugin, too!

More on this later!

Hosted Discord Bot Services, IFTTT

Why reinvent the wheel? There are services out there where you can link your Twitter account to, provide some RSS feed URL’s, and just have them send to your Discord channel. It’s true and I don’t think these servers are constrained by rate limiting from Discord either (more on that in the next section)

My issue is that, whether it be IFTTT or Pingbot, there are limitations to both free and paid subscriptions. There is also the element of security and privacy. Integrating Twitter with these services requires you link your Twitter account to their applications #ThirdPartyBreach?

I understand I’m not dealing with sensitive information here, it’s just more about principle. I’m not paranoid, you are!

Below is a high-level list of some limitations I found with hosted/Discord bot services:

Pingcord:

• Free account limits 1 Twitter integration, up to 3 Twitter handles (no Twitter List support) and can only send to one Discord channel
• Premium provides 4 additional integrations per service
• Can add more integrations than the maximum 5 with premium by buying additional credits

IFTTT:

• Free account offers up to 3 Applets
• Twitter integration is decent, can create based on Tweets by user, by search, by hashtag, etc.
• Unlimited Applets with a Premium subscription

Zapier:

• Free account offers up to 5 Zaps
• Supports Twitter Lists!
• Starter account allows up to 20 Zaps @ $20.00/mo

This list of hosted services is by no means exhaustive. With all that said, what about self-hosting options?

Self-hosted Solution using Huginn

Huginn is an awesome platform for automating a lot of different web-related tasks, including things such as pulling Tweets, formatting and parsing RSS content, and a lot more. You can self-host Huginn on a VM or, in my case (which is easier to setup having tried the self-hosting method), Docker!

What I like about Huginn in particular is this one liner from the GitHub README.

Think of it as a hackable version of IFTTT or Zapier on your own server. You always know who has your data. You do.

Which was important to me as I outlined points of concern with hosted-services like IFTTT and Discord bot services.

There’s a slight learning curve to anything within Huginn, but the RSS portion is pretty straight forward. You can organize your “agents” into “scenarios” which feed off of one another; a logical if/then/else type flow.

Organizing agents within “Scenarios”

Agent Flow

The “Agent Flow” screenshot above illustrates what each agent is doing and what its next step is in a flow. In my case, I use:

• RSS Agent - pull data from a given RSS feed
• Formatting Agent - extract the fields I care about (title, URL, description, etc.)
• Post Agent - send formatted data via HTTP POST somewhere - i.e.: Discord Webhook URL

Aww, yeah!

Besides the RSS component, you can use Huginn to pull Tweets from Twitter’s API. Unfortunately, the configuration for connecting Huginn to Twitter is rough, but doable. There’s some requirements to expose Huginn to the internet for Callback URL purposes.

You can also take a look at logs from your Post Agents. This is when I started to think Discord may not be the right approach. Inspecting Post Agent logs for these RSS agents shows some posts don’t go through due to rate limiting!

This means I could miss out on a potentially important post from an RSS feed. You can, however, stagger how often the Huginn Post Agent will send to the Discord webhook in a given timeframe, but this just slows things down and doesn’t solve the problem.

Huginn has been great for the RSS component, but the Twitter integration is steep. Regardless, I would take a look at what Huginn has to offer with their Agents (integrations) because there are a lot of potential use cases!

What about monitoring Twitter?

In order to leverage Twitter’s API for things like, well, pulling Tweets from accounts of interest, you need to submit an application and go through approval by Twitter in order to be granted a developer account. For reference, a developer account provides you with access to Twitter’s API.

Now, this is a big one. There are plenty of threat intelligence related Twitter accounts that actually provide decent insight and, well, intelligence. This is almost the linchpin for this Discord CTI platform. Unfortunately, it’s not that easy to achieve.

You need to justify to Twitter why you need a developer account. Which is fine, I get it: abuse of the API, misinformation, bots, spamming, and all that good stuff.

The conversation with their review staff has gone something like the following so far:

Twitter: “Why do you need this?”

mgior: “I’d like to leverage the API to pull in Tweets from select Twitter accounts that publish useful CTI that I would like to bring into a channel on a Discord server I own/maintain solely, dedicated to the consumption, review, and research of CTI as well as to have a single pane of glass to gather this information in.”

Twitter: “But, like, why do you need it?”

mgior: “There are feeds on Twitter, from many Twitter accounts, that publish helpful information in regards to emerging threats and vulnerabilities, for example. Leveraging the API to be able to pull in such information would help in better understanding those emerging threats, aide in decision-making… and make for a fun project.”

Twitter:

I essentially broke out a Kanban board to create “user stories” to continue my justification to Twitter.

Either way, I was finally approved for a Developer Account after providing more detailed “user stories”

Getting Tweets into Discord

Welp, using webhooks would have been helpful here, until I learned that was an Enterprise/Premium Twitter API feature, so that’s out.

Not a problem though, I can just use some Python to make calls to the API to get the hashtags, users, and other context I’m looking for, which is fine! The issue still remains on how to get that information into Discord.

There is a Discord Python API wrapper that I found on GitHub called discord.py, but I also came across something more to the point, discord-twitter-bot which looks promising for my use case; it even support Twitter Lists - more refinement!

It’s working!

The YAML configuration for Discord-Twitter-Bot is very in-depth, but in a good way! Take a look at some examples here in the documentation.

In the screenshot above, I’m testing Discord-Twitter-Bot in a very basic way by having it monitor a single List and send any Tweets within to a channel on my Discord server via webhook. I also tried one of the examples from the documentation where it simply monitors for any mention of the word ‘python’ and I can’t be certain, but by the amount of messages in the channel, I can’t say that Discord is rate limiting it.

One issue I found when using Twitter Lists that doesn’t seem to be documented, is that you can’t simply copy/paste the List URL from your browser into the YAML configuration file.

For example, if I created a List you would be able to find it by browsing to https://www.twitter.com/mgior/lists If my list was called “CTI Stuff”, you would need to enter the List URL into the YAML configuration file as https://www.twitter.com/mgior/lists/CTI-Stuff (note the - for spaces) else Discord-Twitter-Bot will run, but you’ll see the following error:

Bot started.
[{'code': 34, 'message': 'Sorry, that page does not exist.'}]

Based on my limited testing so far, I would definitely say that Discord-Twitter-Bot is a self-hosted option I’m going to keep on the table! There’s some homework to do in regards to the configuration possibilities for fine-tuning, but getting this working was very low-effort.

If I experience some limitations with integrating Twitter into the Elastic Stack (which I will) I can always come back to this and point the webhook to a Logstash input!

Maybe I’ll do some code review, too! Just in case. It came from GitHub, ya know!

Getting Tweets and RSS Content into Elastic Stack

As I mentioned earlier, using the Elastic Stack, we can ingest both Tweets and RSS feed content through the power of Logstash. Logstash has input plugins for both RSS and Twitter.

Assuming you have an Elastic Stack setup, installing the Twitter and RSS plugins is as easy as:

cd /usr/share/logstash
sudo ./bin/logstash-plugin install logstash-input-twitter
sudo ./bin/logstash-plugin install logstash-input-rss

Obviously, you will need to create some Logstash configuration files to match. Below are example configurations for both Twitter and RSS inputs.

input {
  twitter {
    consumer_key => "XXXX"
    consumer_secret => "XXXX"
    oauth_token => "XXXX"
    oauth_token_secret => "XXXX"
    follows => ["123","456","789"]
    keywords => ["malware","ioc","apt","ransomware"]
    full_tweet => true
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "twitter-%{+YYYY.MM.dd}"
  }
}

input {
  rss {
    url => "https://www.zdnet.com/topic/security/rss.xml"
    interval => 300
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "zdnet-rss-%{+YYYY.MM.dd}"
  }
}

Unfortunately, it doesn’t look like Lists are supported using the Logstash plugin. In fact, the Twitter Logstash plugin is somewhat limited based on the documented configuration options - but not the end of the world!

The key configuration parameters that we would get use out of would be the keywords and follows options.

The keywords option can be configured as an array; i.e.: ["cybersecurity","ransomware","cobaltstrike"] which can be somewhat impractical and annoying to manage and likely generate a lot of noise. However, if we include the follows option, we can hone in on specific keywords in conjunction with the Twitter ID’s we specify using the follows option to reduce some of that noise - or don’t use the keywords option at all if we’re just want to consume content from the Twitter ID’s we care about.

With that, to get more granular, we can leverage the follows option which can also be configured as an array containing the Twitter ID’s of the Twitter accounts that we are interested in pulling from. To obtain the TwitterID of a particular account, there are tools available online, such as this one, where you plug in the Twitter username and it will fetch the ID for you. Similar to the keywords option, these can be placed in an array as well, i.e.: ["123","456","789","..."]

Dashboard, of course… it’s a work in progress.

Retrospective & ToDo’s

Since I’m starting to get the data I’m interested about, I still need to do some work to massage it and make it a little more actionable:

• Logstash filter on Tweets to carve out data such as: CVE ID, file hash, domain name, IPv4 addresses, APT mentions, product names
• Dashboard the above items

Conclusion

I hope this (lengthy) post provided some insight into the different approaches available to send different types of data to Discord or an Elastic Stack. There are many hosted services/bots and self-hosted options that fit the bill and have nearly endless possibilities and integrations; especially Huginn.

Whether or not this is practical, it has been useful for me when it comes to research and just staying on top of trends, IOC’s, and security news.

I’m a nerd and the painful, customized route (with dashboards, of course) was more fun for me and still provided me with a solution I was looking for.