Using BeautifulSoup to grab all the HTML between two tags

Question

I have some HTML that looks like this:

Title

//a random amount of p/uls or tagless text

 Next Title

Answer 1

Here is a complete, up-to-date solution:

Contents of temp.html:

Title
hi
//a random amount of p/uls or tagless text
 Next Title

Code:

import copy

from bs4 import BeautifulSoup

with open("resources/temp.html") as file_in:
    soup = BeautifulSoup(file_in, "lxml")

print(f"Before:\n{soup.prettify()}")

first_header = soup.find("body").find("h1")

siblings_to_add = []

for curr_sibling in first_header.next_siblings:
    if curr_sibling.name == "h1":
        for curr_sibling_to_add in siblings_to_add:
            curr_sibling.insert_after(curr_sibling_to_add)
        break
    else:
        siblings_to_add.append(copy.copy(curr_sibling))

print(f"\nAfter:\n{soup.prettify()}")

Output:

Before:

 
  

   Title
  
  

   hi
  
  //a random amount of p/uls or tagless text
  

   Next Title
  
 


After:

 
  

   Title
  
  

   hi
  
  //a random amount of p/uls or tagless text
  

   Next Title
  
  //a random amount of p/uls or tagless text
  

   hi