Joshua Goode: I think it’s pretty cool to see what people can do with the platform that gives them the flexibility. It gives them a strong base to build upon. And I think Newspack is a shining example of taking things to the next level.

Claudiu Lodromanean: With elections, they’re so infrequent and each one is different. So you have to prepare for the worst, to prepare for so much potential traffic.

Joshua Goode: One of our goals that we are regularly working on and are mostly achieving is that we want to take care of all the technical burden for hosts, so that way they can focus on their product and serve their hosting customers.

Claudiu Lodromanean: We like to say we’re a shared service, not an agency. So we’re not out there building bespoke software for each publisher. We are trying to build things in a general purpose way that works for all the organizations on the platform. Welcome to Impressive Hosting with Jesse Friedman. I am not Jesse Friedman. I’m Joshua Good. At the time of this recording, Jesse is on a sabbatical and graciously asked me to take the reins of the show while he’s away. So for the next however many episodes, I’ll be attempting to fill some impressive shoes. If this is your first time listening to the show, we’re dedicated to uncovering the core tenets of what makes great and dare we say, impressive WordPress hosting as well as the greater ecosystem around it. Before we dive in, be sure to check out impressivehost.com. That’s where you can find and comment on previous episodes. Ask follow-up questions, submit questions for future guests, and you also find all of our links to like, follow and subscribe wherever you listen. Today’s guest Claudiu Lodromanean. He’s going to fill us in a little bit more on Newspack. He was in our previous episode and we kind of left on a cliffhanger on a few things. So we brought him back here. We’re going to delve into some more technical items, maybe a little product stuff. Again, Newspack is the platform and an impressive platform out of that for news empowering news orgs by giving them the tools and service to thrive. And Claudiu has been working on Newspack since the beginning. He has helped architect develop and launch and scale the platform, so he is the perfect person. Based on the last conversation I just had with him, he’s the best one to fill us in on how they deliver impressive hosting. Thanks again for joining us for another episode. We kind of left on some, we kind of got an idea of what Newspack was in the previous one, and we left on some opportunity to delve into the technical world a little bit more. Maybe a little bit of product stuff too. I think first if you want, reintroduce yourself, Claudiu.

Claudiu Lodromanean: I lead engineering at Newspack. Been doing it since Newspack started. Newspack’s about six years old now. It’s grown pretty rapidly. Now we host over 330 organizations all over the world, all different sizes, all different business models. The main thing they got in common is they’re trying to get the news out there.

Joshua Goode: Yeah, that’s perfect. Perfect. So again, as I said in the last episode, we really covered what Newspack is and what they’re doing over there and all the cool stuff. And I got to slip in a little bit about WP Cloud. Not trying to make this a WP Cloud ad, but I think it’s pretty cool to see what people can do with the platform that gives them the flexibility. It gives them a strong base to build upon. And I think Newspack is a shining example of taking things to the next level and really building out not just their product, but really customizing and addressing their hosting to fit the needs for a very specific audience that can also have very demanding sites, definitely around traffic. So I kind of hinted at in the last episode that we may get into a little bit about your prep and the work preparing for the 2024 US elections. And I guess you’re probably dealing with elections all around the world with some of these other organizations. But let’s focus on that for a bit. So tell me, I think you had mentioned you were preparing for six months or so ahead of the 2024 elections and really tried to make sure your partners and hosting were ready for that. So why don’t we delve into that and I guess to start, what does that level of prep actually look like? What are you doing? What are you worried about? What are you trying to address?

Claudiu Lodromanean: Yeah, I mean, it’s a lot. Elections in general, big. The sites have to stay online. Their reputation is on the line, our reputation is on the line. It is one of the things where it is so important that people can visit the site. It works. You’re getting timely information. So we started at least six months beforehand. With elections, they’re so infrequent and each one is different. So you have to prepare for the worst, to prepare for so much potential traffic. We have no idea. Is this site going to get 10 times the regular amount of traffic on election day? Is it going to get 30 times? Back in the 2020 election, Bangor Daily News is one of the publishers hosted with us and they were the first to call the election. It was 10 times the traffic. It was sort of a breaking news event.

Joshua Goode: Yeah, and you know, all at the same time, you obviously have the national elections in the United States for example. You have these national elections going on in some years. Obviously very big, very important, very big to a lot of people. But you also are dealing with a lot of people who are checking these sites for just local elections. So you’re getting a surge of traffic for all these different things all the way down. I was checking some of my local news sites for just my local city council election, so you’re getting this massive combination of traffic to all kinds of different posts and pages, all kinds of different things that are tracking those.

Claudiu Lodromanean: Actually, if I could jump in, those are actually more important and more what people are reading these sites for than the national election results. Those are all over TV, all over the giant websites. But for mayoral races, for ballots, all of these citywide and statewide things, this is where people go to find that information.

Joshua Goode: Right. Right, right. So I guess let’s go into some of the prep work of that. What are some things you discuss from the base layer? In our previous episode, we talked about how important database optimization is and plugin optimization and making sure caching is good. So I imagine some of that’s carrying over into some of this prep, but is there anything really notable or specific? What are you actually doing in these six months leading up? What are you doing for the platform, but maybe considering specific things for specific partners? What are some examples of what that looks like?

Claudiu Lodromanean: Yeah. So ultimately it’s the same as everything. Our goal is to prepare all the sites should be as prepared for traffic as if every single day was election day. Because you never know. Whether it was a political assassination in Haiti a while back, and that’s not something you can prepare for. Or an earthquake in BC and one of our sites was getting a hundred thousand requests a minute. But our goal with the election was to get all of the sites to that point, so that we will be set for election and indefinitely going forward.

Joshua Goode: Yeah. That way, any viral posts that take off or anything like that? Yeah, I can see that. There’s definitely from time to time my local city shows up in national news and it’s always interesting who breaks that thing, who causes some tweet or some post somewhere that gets shared and starts driving all that traffic out there to the news site that broke it first.

Claudiu Lodromanean: The first step is monitoring, alerting, examining the server charts and the server graphs. Because first you have to identify where are the problems. Is it across all the sites or is it just a handful of sites? Step one was analyze all of the server data that we can to identify where any performance problems are or where any caching issues are.

Joshua Goode: Yeah, because I can see you have your product and you have these common plugins and configurations that you’re putting out, and so you may be able to find something in that that will hopefully help all sites. But I imagine you have hundreds of these organizations that they’re all going to have something unique about them in some way. They’re doing their own thing. So that sounds like a lot of work of making sure that they’re all hopefully coming up to the same page, even though you have so many unique sites.

Joshua Goode: Mm-hmm.

Claudiu Lodromanean: problems is it across all the sites or is it just a handful of sites? Step one was analyze all of the server data that we can to identify where any performance problems are or where any caching issues are, things like this.

Joshua Goode: Yeah, because I can see you have your product and you have these common plugins and configurations that you’re putting out, and so you may be able to find something in that that will hopefully help all sites, but I imagine you have all, you know, hundreds of these organizations that they’re all going to have something unique about them in some way. You know, they’re doing their own thing. So that sounds like a lot of work of making sure that they’re all kind of hopefully coming up to the same page, even though you have so many unique sites.

Claudiu Lodromanean: Yeah. No, it is. Our goal with all this is to keep the tech stack as standardized as possible. That’s the only way to effectively do this across, you know, more than 300 sites at this point. We’re a small team. We like to say we’re a shared service, not an agency. So we’re not out there building bespoke software for each publisher. We are trying to build things in a general purpose way that works for all the organizations on the platform. And that’s key to being able to find a performance issue, fix it, and it helps 50 publishers.

Joshua Goode: Yeah. Yeah, that’s smart.

Claudiu Lodromanean: Being WordPress, you do have a number of people that have this one-off weird plugin that we then have to debug occasionally.

Joshua Goode: Yep. Yeah, I can imagine that pops up. So you’ve built some tooling, you mentioned server logs and everything else, but you’ve built some specific tooling, right? To query all of your sites, see what plugins or features are actually in use, so that maybe you can get a little tip on where you need to start with some of the debugging. If that’s correct, has that given you good insight?

Claudiu Lodromanean: We have tooling that we can query across all sites, like which sites are doing a subscriptions type of revenue model, which sites are using MailChimp for their email service provider. Because you know, everything is built in a general purpose way, but different organizations use different parts of it. So that’s important to being able to find who is affected by something, and how many sites do we need to roll this out to?

Joshua Goode: Got it, got it. Right.

Claudiu Lodromanean: Yeah. So on that, I mean, has there been anything, and we don’t have to name too many specific names or anything, but has there been anything specific that’s been surprising or a bit of a pain or something that y’all were able to fix and then, you know, it helped all of the sites and maybe even non-Newspack sites. Is there anything like that that’s popped up recently?

Claudiu Lodromanean: Yeah, so one of them, during the election actually that we discovered as we were analyzing things was sitemaps. So if you go to the previous episode, I was saying that some of these sites have hundreds of thousands of articles. That presents unique scalability challenges. We discovered when a bot was going to the sitemap, it would query the whole database of 100,000 articles and build the sitemap dynamically. So we implemented kind of a static sitemaps approach, so we generate them asynchronously. It’s a static file. It causes no server load when it’s accessed. And that was a huge improvement in terms of CPU usage and things like that.

Joshua Goode: Yeah. Yeah, I can imagine. That’s cool. That’s a nice little finding. So that was discovered during your election prep? That’s when that was kind of discovered.

Claudiu Lodromanean: Yeah, election prep. One of the things that we determined, you know, we noticed as we were analyzing the server data is sitemaps would take 15 seconds to generate and it causes a spike of 10 CPUs whenever someone visits them. We should address that.

Joshua Goode: Right. So moving on from the sitemap thing, I think that’s pretty cool. In a previous episode, you mentioned something about using Elasticsearch. Break that down for a bit. Is that because I know search and I assume that you’re using this to address searches on these news sites, which search for me and analyzing hundreds and hundreds of sites myself, search can sometimes be a very cumbersome thing. It can also use a lot of CPU resources. There are different approaches you can take for a fast, low resource consuming search. I’m guessing that’s what you’re doing with Elasticsearch in this case.

Claudiu Lodromanean: Actually not quite, not for search. We do use Jetpack search, like the Jetpack search works great. It just replaces WordPress search. It’s super fast. We’re using it for complex filtering, so you go to a news site homepage and there’ll be, you know, a section on the top that’s featured and then latest stuff in this category, latest stuff in this category. And there’s all sorts of complex filtering and deduplication behind the scenes. This generates a really gnarly MySQL query, which especially if you have a hundred thousand posts in the database and you need to find four posts from the last week that have two categories, but not this tag, then it’s not super scalable at all. So we actually use the same Jetpack Elasticsearch index from the search, but we hook into it. We use that to render different sections of the homepage, which is a huge scalability boost compared to doing it with SQL.

Joshua Goude: Yeah. No, that’s big. Yeah, that’s big. I appreciate that explanation there. Because yeah, I was not expecting that. But that totally makes sense because search can be expensive depending on how you’re approaching that. Anytime you’re filtering content in that way. So we’ve covered a bit of your election prep. We’ve covered this pretty cool usage of Elasticsearch there. Some of your tooling that you use to identify what sites are doing, what sites are using, so you can narrow down and see if maybe a specific site needs some help due to some unique configuration. Maybe there’s a broader thing that you can do to fix everything else. I imagine you’re probably solving for a lot of other things. And a big thing that comes to mind is security. So you’re kind of like the de facto security team, right? For these publishers.

Claudiu Lodromanean: Yes, because you know, as I think I’ve said in the last episode, we essentially are the publisher’s technical team. More than 90% of them don’t have any technical person on staff, no developers on staff. So it’s kind of on us to, one, make sure their sites are secure, and two, educate them around best practices. So making sure that their staff has two-factor authentication to sign in to the site. Monitoring and protecting against card testing attacks. News sites in general, they’re big targets for a few reasons. You know, they have a ton of reach. A lot of people read these, so spammers want to get their stuff on there for SEO and for the eyeballs. As well, they have a lot of e-commerce features, so abilities to donate, sign up for a membership, to buy merch, and this makes them a target for the more e-commerce type of security issues. So card testing attacks, card stealing attacks, any sort of this kind of thing. We have to make sure the sites are secure and stable across all fronts.

Joshua Goode: Yeah. Yeah. So everything from, you mentioned two-factor authentication, I guess, you know, strong passwords even if you have multi-factor authentication and stuff there. You know, any other kind of specific things, anything unique that y’all are doing or any, I guess just, what’s something that seems to help a lot in terms of the protections that you’re doing for these sites.

Claudiu Lodromanean: So the two-factor authentication is the biggest. I cannot overstate that. 99% of hacks are not someone was super clever and found an exploit and got into the site that way. It’s like you used this password somewhere else and they tried it here and it worked. You know, if you get that sorted, you are solid. Add reCAPTCHA to any form, newsletter signup form, tip form, contact form, these types of things, you know, it’s standard stuff, but it’s very important to just have the fundamentals in place 99% of the time.

Joshua Goode: Yeah. Perfect. Perfect. So I guess yeah, and you’re just including a lot of this, you’re setting this up for people by default. You’re not leaving this to chance again, you’re their security team, you’re their technical team. So you’re providing that foundation by default, and I guess you’re doing some education as well, helping them avoid security mistakes along the way.

Claudiu Lodromanean: here and it worked.

Joshua Goode: Right.

Claudiu Lodromanean: You know, if you get that sorted, you are solid. Add reCAPTCHA to any form, newsletter form, tip form, form type of things. You know, it’s standard stuff, but it’s very important to just have the fundamentals in place.

Joshua Goode: Yeah.

Claudiu Lodromanean: You of the time.

Joshua Goode: Perfect. Perfect. So I guess, yeah, and you’re just including a lot of this, you’re setting this up for people by default. You’re not leaving this again, you’re like their security team, you’re their technical team. So you’re providing that foundation by default, and I guess you’re doing some education as well, like kind of helping them avoid security mistakes along the way.

Claudiu Lodromanean: Yeah, no, we do provide guidelines. These are, you know, the things that your team should be doing in order to be safe. And, you know, it’s important because some of these are sites, you know, we have publishers doing reporting where their own government is hostile towards them. In the last election, to bring it back to that, one of our publishers is the Haitian Times and after the eating the cats and dogs thing, like their site was under attack just nonstop for two months.

Joshua Goode: Hmm. Yeah. Yeah. So at the WP Cloud level, you know, we do provide some protections, but ultimately it comes down to how sites are configured, how the users are using them. It sounds like News Pack is taking great efforts to make sure that all these sites have a very baseline kind of security package there and all the guidelines to help resist and be secure against lots of attacks like that.

Claudiu Lodromanean: Yeah. And one of the other things that has been really helpful towards this is like all the Jetpack security features. So brute force protection, malware scanning, it can, you know, detect and alert if someone manages to sneak some malware onto the site. So we, you know, that’s fundamental and we don’t have to necessarily do anything special. We just use it and it works great.

Joshua Goode: Yeah, yeah. Jetpack, you know, also has the SSO, they have, and kind of tied to Jetpack, but really its own independent product, Akismet. So I imagine Akismet is probably a big part of the package.

Claudiu Lodromanean: Yep.

Joshua Goode: Cool. Cool. So kind of stepping away from that, but keeping some of that in mind a bit and kind of going back with, you know, maybe why news organizations should consider News Pack. Maybe they aren’t right now. So if someone was running a news organization on a generic WordPress hosting right now, so let’s just say it’s just a standard shared hosting right now, not necessarily impressive hosting right now, you know, what’s one thing that they’re probably underestimating about their needs, like not really considering.

Claudiu Lodromanean: So, you know, we have a bunch of case studies on the website if you want to poke around it to see the sorts of improvements people have made coming from generic WordPress site to News Pack. Our features are really designed to save publishers time and to make them more money. Our, we have dynamic segmentation built in that’s, you know, not something that you can get out of an out-of-the-box plugin. So you can send targeted messaging to people based on if they’re a donor, if they’ve signed up to the newsletter, really fine-tune the funnel and the people that do it well, you know, they essentially double their reader revenue or more over time. You can compose your newsletters and send them directly with the block editor. You know, people save hours a week doing it this way. We’re really focused on measurable results.

Joshua Goode: Yeah. Yeah. So yeah, really taking all this technical burden away and allowing them to focus on their service, their news, their relationships with their readers, and kind of growing and thriving there. Similar, you know, with WP Cloud. Obviously News Pack has taken it and really customized it, done a lot of the very specific things, technical-wise with your product and so on. But a big part of WP Cloud and why we try to kind of make it a situation where we do a lot of the baseline technical things for you. Now, obviously, you know, news sites are very unique and very high demand, and we’ve actually made some improvements to the platform based on things we’ve learned from your team and the partners you work with. But ultimately, one of our goals that we are regularly working on and are mostly there serving, is that we want to take care of all the technical burden for hosts, so that way they can focus on their product and then serve, you know, their hosting customers. So we do have a bit of a trickle-down thing of these goals that are aligning quite a bit there. And it sounds like it’s working overall, it sounds like we’re, yeah.

Claudiu Lodromanean: We just got a systems guy on our team maybe a month ago, but for the previous six years, we did not need a systems person. We did not need an infrastructure person. WP Cloud just handled it really well out of the box.

Joshua Goode: Yeah. Yeah. That’s really cool. Going back to, you know, considering traffic and big high-profile events that these news sites are just going to get. It’s not a matter of if, it’s a matter of when, when a piece of news is gonna get viral. Maybe when a scraper decides to start hitting their site really hard. We’ve talked about logging, we’ve talked about mitigation and planning and stuff like that. Ultimately, some things will fall through the cracks. What kind of, you know, alerts do you have? Do you have a bunch of alerting set up for this? What are some specific events that you’re looking for with that alerting as well?

Claudiu Lodromanean: Yes. We have a number of alerts. You know, at the most basic one is we have a service that pings the site every few minutes and it throws a big alert text to my phone and everything. If it gets a status code that’s not 200, so if it’s a 500 or if it’s a 404 or you know, 503, like anything other than 200, it’ll ping the whole team. We also have alerting for CPU usage or for card testing attacks. And these are more proactive. You know, we don’t want to respond to things when the site is down. We want to catch it before it goes down so we can prevent it in the first place. So we get Slack alerts if CPU usage on a site goes to an uncomfortable level.

Joshua Goode: Nice. Nice. And I guess we can kind of wrap up some of the technical talk a little bit, but kind of bringing together everything we talked about in terms of optimizations and crawlers and CPU usage here. I know probably crawlers, scrapers, those can definitely be demanding if the pages are uncached. We have talked about, you know, the need to optimize certain plugins and database queries, but has there been anything noticeable that you’ve found that used a lot of CPU unexpectedly. So have you examined some sites here recently or any time that, and you found, I know you mentioned sitemaps was what’s a big one prior to the election, but is there anything else, since you brought up CPU again? So is there anything else like that?

Claudiu Lodromanean: Yeah. I mean, this may be too in the weeds, but it’s something I was dealing with earlier this week, which is, if you’re using WooCommerce, by default, it has that high-performance order storage setting enabled. So anytime something happens with an order, it will also like do a bunch of database writes to make it also synchronize to the legacy order system. And, you know, especially on high-traffic e-commerce sites, like, or in our case sites that have a lot of members and subscribers, that causes a bunch of CPU load. If you’re using all modern WooCommerce plugins, they have the compatibility. It’s not needed. You know, we toggled that off. It was instant drop of like 50% CPU usage and just like perfect stability.

Joshua Goode: Yeah. Yeah. So that, yeah, that’s exactly something, a story I was looking for there. And, you know, it’s a challenge, WooCommerce, which I believe you worked on WooCommerce for some time before News Pack. Is that right?

Claudiu Lodromanean: I did. Yeah. Yeah. You know, about two and a half years before News Pack existed, I was on the WooCommerce team working on the plugin and we use it for all of our e-commerce capabilities. It’s like it doesn’t make sense for us to re-implement it all because it’s a rabbit hole. You get into tax calculations, inclusive tax, exclusive tax, different gateways like it’s a solid foundation, we use it.

Joshua Goode: Yeah, yeah. And that specific issue you brought up, you know, that’s a challenge of WooCommerce has to solve so many things at once, and it also has to solve for the fact that not all plugins are up to the newest, greatest performant order storage there. So, but that’s definitely a nice find and that’s kind of the tips and things I’m looking forward to getting into in other episodes and stuff too. So, definitely never, if we do another episode and stuff, never feel, never feel held back, or excuse me, never hold yourself back in terms of the technical stuff. We can, I think that’s a really cool example.

Joshua Goode: Awesome.

Claudiu Lodromanean: It doesn’t make sense for us to re-implement it all because it’s a rabbit hole. You get into tax calculations, inclusive tax, exclusive tax, different gateways. It’s a solid foundation, so we use it.

Joshua Goode: Yeah, yeah. And that specific issue you brought up, that’s a challenge WooCommerce has to solve so many things at once. It also has to solve for the fact that not all plugins are up to the newest, greatest performant order storage. But that’s definitely a nice find and that’s the kind of tips and things I’m looking forward to getting into in other episodes too. So definitely, if we do another episode, never hold yourself back in terms of the technical stuff. I think that’s a really cool example.

Claudiu Lodromanean:

Joshua Goode: As we kind of wrap up the show, I’ve got one question. It’s not really completely dedicated to just Newspack and news organizations, just looking at the broader WordPress hosting ecosystem. One thing in the show that we talk about is really identifying the core things of what makes impressive hosting, what makes for a good user experience. And just trying to package all of that together to hopefully have these conversations with people to elevate the bar, elevate the minimum expectations. Because I think for WordPress, for WooCommerce, ideally, I want people to be on Newspack. I want people to be on WP Cloud. But at the end of the day too, just thinking about WordPress, WooCommerce and so on, that whole ecosystem, I think for its ultimate success, its continued growth and continued thriving, we really want to see hosts be better and reach a higher standard. So with that in mind, and looking at the broader WordPress hosting ecosystem, what should hosts, or what should more hosts be thinking about that they’re probably not, or maybe what are some things that they should be doing that they may not be doing?

Claudiu Lodromanean: Yeah, so I think observability is a big one. The ability to analyze and determine which IP addresses are hitting my site, which bots are hitting my site, what requests have slow performance. And then even within that, what part of this request is this issue coming from? What hook or filter or plugin is causing problems? At least for these more enterprise-scale things, that’s super important.

Joshua Goode: Perfect. Perfect. Alright, Claudiu, thank you so much for joining me again. So folks can learn more about Newspack over at Newspack.com. Is that right?

Claudiu Lodromanean: Yes, absolutely. Newspack.com. Lots of information there.

Joshua Goode: Right, and where’s the best place to find you?

Claudiu Lodromanean: Find me on LinkedIn, Claudiu Lodromanean. I’m gonna be the only Claudiu Lodromanean on there.

Joshua Goode: Nice, nice. And of course you can find me at wp.cloud if you’re interested in what we’re doing over there. If you’re just looking for me personally, goode, G-O-O-D-E dot PRO and of course you can find this episode and more over at Impressive Host and we’ll let Jesse kind of do the roll credits on all the other places to find us. But definitely check out the site and get all the links that you need to find this episode and more.

Claudiu Lodromanean: Thanks for having me, Josh.

Joshua Goode: Yeah. Thank you.