I recently came across the article “The @Dharmesh Test: 16 Questions For Better SaaS Companies“. I worked on one very cloud system, the Azure AppFabric. So I started asking myself these questions about the AppFabric, even though it is actually a PaaS, not SaaS. Don’t expect any secrets about the services, it is all from public knowledge.
1. Is there exactly one version of your software that services all users?
Absolutely. There is one code base for all.
2. Do you have a free trial
No, not in production. I’ve been putting pressure on the marketing team to at least provide limited free additions to digerati and educators. Other than the production release, there is a free version for the Labs/CTP release, which is a non-production environment with experimental features.
3. Do you track key metrics like CAC, LTV, and cancellation rate?
I won’t say what is and isn’t being tracked, but I will say that the emphasis isn’t on making money for Microsoft, but it is for ensuring the best performance and highest uptime.
4. Are your prices published on your website?
Yes, but frankly pricing is incredibly cryptic. I challenge you to understand the AppFabric pricing model for the Service Bus (the Access Control Service is strait forward). Nobody can really explain it very well. I believe this is the teams greatest failure, as the pricing model was designed to enable the services to be measurable and scalable for the team, not for the customer.
5. Can people start paying you (become customers) without interacting with a human?
Absolutely. Though the sign-up process requires like 20 steps. It’s a work in progress.
6. Do you make more money from users that are getting more value?
No, there is exactly one consumption/price ration and no special features. This is probably a win because it would further complicate the already complicated model.
7. Does your application have a simple, clean API?
N/A. AppFabric is a PaaS, not SaaS, so the whole thing is an API. I would consider it simple though, it is designed for the advanced developers. The ACS service though is working hard to making the APIs much simpler and cross-platform compatible.
8. Can customers export their data out of your system?
Yes
9. Can you deploy (and rollback) a new version of the software in one-step without downtime?
Yes. A large chunk of engineering efforts for building AppFabric focused on building the operations, not the technology itself. The deployment process is incredible.
10. Is there an online community where users can interact with your team and with other users?
Yes. There are forums, email aliases to the entire team, and we even have dedicated “field” people to work with customers.
11. Do new developers commit code on their first day at work?
Theoretically they could if the check-in is small enough. There are a lot of habits new developers have to get used to, including code reviews, the check-in process and tools, etc. There is a learning curve, but it’s a good learning curve as engineering practices are top-notch.
12. Do you release code to production at least once a month?
No, but it goes out every 2-3 months. Still not bad.
13. Do you maintain a centralized backlog of new product ideas, bugs, and issues?
Damn right we do. The Feature PMs (my role) is responsible for all that goodness.
14. Do you regularly run A/B tests?
N/A I think this is for SaaS moreso than PaaS. We just talk to customers to see what they have to say.
15. Do you invest in design and user experience?
Yes, but not enough. One of the great challenges about working on PaaS is that it is designed by developers for developers, so it is incredibly technical. There are few people that care about UX. I happen to be one of the people that does care and I personally helped make some of the improvements in the Portal.
16. Does your company or product have a personality?
This is the squishiest of the questions, and I don’t know how to answer that.
