# Script to download multiple URLs from a text file with improved line handling
# Usage: ./download_urls.sh urls.txt [output_directory]

# Check if input file is provided
if [ -z "$1" ]; then
    echo "Error: Please provide a text file containing URLs"
    echo "Usage: $0 <input_file> [output_directory]"
    exit 1
fi

input_file="$1"
output_dir="${2:-./downloads}"

# Check if input file exists
if [ ! -f "$input_file" ]; then
    echo "Error: Input file '$input_file' not found"
    exit 1
fi

# Create output directory if it doesn't exist
mkdir -p "$output_dir"

# Read and process valid URLs into an array
urls=()
while IFS= read -r line || [[ -n "$line" ]]; do
    # Trim leading/trailing whitespace and remove CR characters
    trimmed_line=$(echo "$line" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' | tr -d '\r')
    
    # Skip empty lines after trimming
    [[ -z "$trimmed_line" ]] && continue
    
    # Validate URL format
    if [[ "$trimmed_line" =~ ^https?:// ]]; then
        urls+=("$trimmed_line")
    else
        echo "Skipping invalid URL: $trimmed_line"
    fi
done < "$input_file"

total_urls=${#urls[@]}

if [[ $total_urls -eq 0 ]]; then
    echo "Error: No valid URLs found in input file"
    exit 1
fi

echo "Starting download of $total_urls files to $output_dir"
current=1

# Download each URL from the array
for url in "${urls[@]}"; do
    # Extract filename from URL or generate unique name
    filename=$(basename "$url")
    if [[ -z "$filename" || "$filename" =~ ^$ ]]; then
        filename="file_$(date +%s%N)_${current}.download"
    fi

    echo "[$current/$total_urls] Downloading $url"
    
    # Download with curl including error handling
    if ! curl -L --progress-bar --fail "$url" -o "$output_dir/$filename"; then
        echo "Warning: Failed to download $url"
        rm -f "$output_dir/$filename" 2>/dev/null
    fi
    
    ((current++))
done

echo "Download complete. Files saved to $output_dir"

news

797 users here now

Welcome to c/news! We aim to foster a book-club type environment for discussion and critical analysis of the news. Our policy objectives are:

• To learn about and discuss meaningful news, analysis and perspectives from around the world, with a focus on news outside the Anglosphere and beyond what is normally seen in corporate media (e.g. anti-imperialist, anti-Zionist, Marxist, Indigenous, LGBTQ, people of colour).

• To encourage community members to contribute commentary and for others to thoughtfully engage with this material.

• To support healthy and good faith discussion as comrades, sharpening our analytical skills and helping one another better understand geopolitics.

We ask community members to appreciate the uncertainty inherent in critical analysis of current events, the need to constantly learn, and take part in the community with humility. None of us are the One True Leftist, not even you, the reader.

Newcomm and Newsmega Rules:

The Hexbear Code of Conduct and Terms of Service apply here.

1. Link titles: Please use informative link titles. Overly editorialized titles, particularly if they link to opinion pieces, may get your post removed.

2. Content warnings: Posts on the newscomm and top-level replies on the newsmega should use content warnings appropriately. Please be thoughtful about wording and triggers when describing awful things in post titles.

3. Fake news: No fake news posts ever, including April 1st. Deliberate fake news posting is a bannable offense. If you mistakenly post fake news the mod team may ask you to delete/modify the post or we may delete it ourselves.

4. Link sources: All posts must include a link to their source. Screenshots are fine IF you include the link in the post body. If you are citing a Twitter post as news, please include the Xcancel.com (or another Nitter instance) or at least strip out identifier information from the twitter link. There is also a Firefox extension that can redirect Twitter links to a Nitter instance, such as Libredirect or archive them as you would any other reactionary source.

5. Archive sites: We highly encourage use of non-paywalled archive sites (i.e. archive.is, web.archive.org, ghostarchive.org) so that links are widely accessible to the community and so that reactionary sources don’t derive data/ad revenue from Hexbear users. If you see a link without an archive link, please archive it yourself and add it to the thread, ask the OP to fix it, or report to mods. Including text of articles in threads is welcome.

6. Low effort material: Avoid memes/jokes/shitposts in newscomm posts and top-level replies to the newsmega. This kind of content is OK in post replies and in newsmega sub-threads. We encourage the community to balance their contribution of low effort material with effort posts, links to real news/analysis, and meaningful engagement with material posted in the community.

7. American politics: Discussion and effort posts on the (potential) material impacts of American electoral politics is welcome, but the never-ending circus of American Politics© Brought to You by Mountain Dew™ is not welcome. This refers to polling, pundit reactions, electoral horse races, rumors of who might run, etc.

8. Electoralism: Please try to avoid struggle sessions about the value of voting/taking part in the electoral system in the West. c/electoralism is right over there.

9. AI Slop: Don't post AI generated content. Posts about AI race/chip wars/data centers are fine.

MODERATORS