Wikipedia entry covers the entire concept of company formation and performance:
The theory of the firm consists of a number of economic theories that explain and predict the nature of the firm, company, or corporation, including its existence, behaviour, structure, and relationship to the market.
Ronald Coase has specific theories around companies and transaction costs.
The model shows institutions and market as a possible form of organization to coordinate economic transactions. When the external transaction costs are higher than the internal transaction costs, the company will grow. If the external transaction costs are lower than the internal transaction costs the company will be downsized by outsourcing, for example.
Today, as communication tools and ways of working remotely become easier and easier, this supports the rise of freelancers and groups of people forming to get work done, because use of these new tools make communication / information / transaction lower, without having to form a company.